diff options
| author | dim <dim@FreeBSD.org> | 2017-09-26 19:56:36 +0000 |
|---|---|---|
| committer | Luiz Souza <luiz@netgate.com> | 2018-02-21 15:12:19 -0300 |
| commit | 1dcd2e8d24b295bc73e513acec2ed1514bb66be4 (patch) | |
| tree | 4bd13a34c251e980e1a6b13584ca1f63b0dfe670 /contrib/compiler-rt | |
| parent | f45541ca2a56a1ba1202f94c080b04e96c1fa239 (diff) | |
| download | FreeBSD-src-1dcd2e8d24b295bc73e513acec2ed1514bb66be4.zip FreeBSD-src-1dcd2e8d24b295bc73e513acec2ed1514bb66be4.tar.gz | |
Merge clang, llvm, lld, lldb, compiler-rt and libc++ 5.0.0 release.
MFC r309126 (by emaste):
Correct lld llvm-tblgen dependency file name
MFC r309169:
Get rid of separate Subversion mergeinfo properties for llvm-dwarfdump
and llvm-lto. The mergeinfo confuses Subversion enormously, and these
directories will just use the mergeinfo for llvm itself.
MFC r312765:
Pull in r276136 from upstream llvm trunk (by Wei Mi):
Use ValueOffsetPair to enhance value reuse during SCEV expansion.
In D12090, the ExprValueMap was added to reuse existing value during
SCEV expansion. However, const folding and sext/zext distribution can
make the reuse still difficult.
A simplified case is: suppose we know S1 expands to V1 in
ExprValueMap, and
S1 = S2 + C_a
S3 = S2 + C_b
where C_a and C_b are different SCEVConstants. Then we'd like to
expand S3 as V1 - C_a + C_b instead of expanding S2 literally. It is
helpful when S2 is a complex SCEV expr and S2 has no entry in
ExprValueMap, which is usually caused by the fact that S3 is
generated from S1 after const folding.
In order to do that, we represent ExprValueMap as a mapping from SCEV
to ValueOffsetPair. We will save both S1->{V1, 0} and S2->{V1, C_a}
into the ExprValueMap when we create SCEV for V1. When S3 is
expanded, it will first expand S2 to V1 - C_a because of S2->{V1,
C_a} in the map, then expand S3 to V1 - C_a + C_b.
Differential Revision: https://reviews.llvm.org/D21313
This should fix assertion failures when building OpenCV >= 3.1.
PR: 215649
MFC r312831:
Revert r312765 for now, since it causes assertions when building
lang/spidermonkey24.
Reported by: antoine
PR: 215649
MFC r316511 (by jhb):
Add an implementation of __ffssi2() derived from __ffsdi2().
Newer versions of GCC include an __ffssi2() symbol in libgcc and the
compiler can emit calls to it in generated code. This is true for at
least GCC 6.2 when compiling world for mips and mips64.
Reviewed by: jmallett, dim
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D10086
MFC r318601 (by adrian):
[libcompiler-rt] add bswapdi2/bswapsi2
This is required for mips gcc 6.3 userland to build/run.
Reviewed by: emaste, dim
Approved by: emaste
Differential Revision: https://reviews.freebsd.org/D10838
MFC r318884 (by emaste):
lldb: map TRAP_CAP to a trace trap
In the absense of a more specific handler for TRAP_CAP (generated by
ENOTCAPABLE or ECAPMODE while in capability mode) treat it as a trace
trap.
Example usage (testing the bug in PR219173):
% proccontrol -m trapcap lldb usr.bin/hexdump/obj/hexdump -- -Cv -s 1 /bin/ls
...
(lldb) run
Process 12980 launching
Process 12980 launched: '.../usr.bin/hexdump/obj/hexdump' (x86_64)
Process 12980 stopped
* thread #1, stop reason = trace
frame #0: 0x0000004b80c65f1a libc.so.7`__sys_lseek + 10
...
In the future we should have LLDB control the trapcap procctl itself
(as it does with ASLR), as well as report a specific stop reason.
This change eliminates an assertion failure from LLDB for now.
MFC r319796:
Remove a few unneeded files from libllvm, libclang and liblldb.
MFC r319885 (by emaste):
lld: ELF: Fix ICF crash on absolute symbol relocations.
If two sections contained relocations to absolute symbols with the same
value we would crash when trying to access their sections. Add a check that
both symbols point to sections before accessing their sections, and treat
absolute symbols as equal if their values are equal.
Obtained from: LLD commit r292578
MFC r319918:
Revert r319796 for now, it can cause undefined references when linking
in some circumstances.
Reported by: Shawn Webb <shawn.webb@hardenedbsd.org>
MFC r319957 (by emaste):
lld: Add armelf emulation mode
Obtained from: LLD r305375
MFC r321369:
Upgrade our copies of clang, llvm, lld, lldb, compiler-rt and libc++ to
5.0.0 (trunk r308421). Upstream has branched for the 5.0.0 release,
which should be in about a month. Please report bugs and regressions,
so we can get them into the release.
Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.
MFC r321420:
Add a few more object files to liblldb, which should solve errors when
linking the lldb executable in some cases. In particular, when the
-ffunction-sections -fdata-sections options are turned off, or
ineffective.
Reported by: Shawn Webb, Mark Millard
MFC r321433:
Cleanup stale Options.inc files from the previous libllvm build for
clang 4.0.0. Otherwise, these can get included before the two newly
generated ones (which are different) for clang 5.0.0.
Reported by: Mark Millard
MFC r321439 (by bdrewery):
Move llvm Options.inc hack from r321433 for NO_CLEAN to lib/clang/libllvm.
The files are only ever generated to .OBJDIR, not to WORLDTMP (as a
sysroot) and are only ever included from a compilation. So using
a beforebuild target here removes the file before the compilation
tries to include it.
MFC r321664:
Pull in r308891 from upstream llvm trunk (by Benjamin Kramer):
[CodeGenPrepare] Cut off FindAllMemoryUses if there are too many uses.
This avoids excessive compile time. The case I'm looking at is
Function.cpp from an old version of LLVM that still had the giant
memcmp string matcher in it. Before r308322 this compiled in about 2
minutes, after it, clang takes infinite* time to compile it. With
this patch we're at 5 min, which is still bad but this is a
pathological case.
The cut off at 20 uses was chosen by looking at other cut-offs in LLVM
for user scanning. It's probably too high, but does the job and is
very unlikely to regress anything.
Fixes PR33900.
* I'm impatient and aborted after 15 minutes, on the bug report it was
killed after 2h.
Pull in r308986 from upstream llvm trunk (by Simon Pilgrim):
[X86][CGP] Reduce memcmp() expansion to 2 load pairs (PR33914)
D35067/rL308322 attempted to support up to 4 load pairs for memcmp
inlining which resulted in regressions for some optimized libc memcmp
implementations (PR33914).
Until we can match these more optimal cases, this patch reduces the
memcmp expansion to a maximum of 2 load pairs (which matches what we
do for -Os).
This patch should be considered for the 5.0.0 release branch as well
Differential Revision: https://reviews.llvm.org/D35830
These fix a hang (or extremely long compile time) when building older
LLVM ports.
Reported by: antoine
PR: 219139
MFC r321719:
Pull in r309503 from upstream clang trunk (by Richard Smith):
PR33902: Invalidate line number cache when adding more text to
existing buffer.
This led to crashes as the line number cache would report a bogus
line number for a line of code, and we'd try to find a nonexistent
column within the line when printing diagnostics.
This fixes an assertion when building the graphics/champlain port.
Reported by: antoine, kwm
PR: 219139
MFC r321723:
Upgrade our copies of clang, llvm, lld and lldb to r309439 from the
upstream release_50 branch. This is just after upstream's 5.0.0-rc1.
MFC r322320:
Upgrade our copies of clang, llvm and libc++ to r310316 from the
upstream release_50 branch.
MFC r322326 (by emaste):
lldb: Make i386-*-freebsd expression work on JIT path
* Enable i386 ABI creation for freebsd
* Added an extra argument in ABISysV_i386::PrepareTrivialCall for mmap
syscall
* Unlike linux, the last argument of mmap is actually 64-bit(off_t).
This requires us to push an additional word for the higher order bits.
* Prior to this change, ktrace dump will show mmap failures due to
invalid argument coming from the 6th mmap argument.
Submitted by: Karnajit Wangkhem
Differential Revision: https://reviews.llvm.org/D34776
MFC r322360 (by emaste):
lldb: Report inferior signals as signals, not exceptions, on FreeBSD
This is the FreeBSD equivalent of LLVM r238549.
This serves 2 purposes:
* LLDB should handle inferior process signals SIGSEGV/SIGILL/SIGBUS/
SIGFPE the way it is suppose to be handled. Prior to this fix these
signals will neither create a coredump, nor exit from the debugger
or work for signal handling scenario.
* eInvalidCrashReason need not report "unknown crash reason" if we have
a valid si_signo
llvm.org/pr23699
Patch by Karnajit Wangkhem
Differential Revision: https://reviews.llvm.org/D35223
Submitted by: Karnajit Wangkhem
Obtained from: LLVM r310591
MFC r322474 (by emaste):
lld: Add `-z muldefs` option.
Obtained from: LLVM r310757
MFC r322740:
Upgrade our copies of clang, llvm, lld and libc++ to r311219 from the
upstream release_50 branch.
MFC r322855:
Upgrade our copies of clang, llvm, lldb and compiler-rt to r311606 from
the upstream release_50 branch.
As of this version, lib/msun's trig test should also work correctly
again (see bug 220989 for more information).
PR: 220989
MFC r323112:
Upgrade our copies of clang, llvm, lldb and compiler-rt to r312293 from
the upstream release_50 branch. This corresponds to 5.0.0 rc4.
As of this version, the cad/stepcode port should now compile in a more
reasonable time on i386 (see bug 221836 for more information).
PR: 221836
MFC r323245:
Upgrade our copies of clang, llvm, lld, lldb, compiler-rt and libc++ to
5.0.0 release (upstream r312559).
Release notes for llvm, clang and lld will be available here soon:
<http://releases.llvm.org/5.0.0/docs/ReleaseNotes.html>
<http://releases.llvm.org/5.0.0/tools/clang/docs/ReleaseNotes.html>
<http://releases.llvm.org/5.0.0/tools/lld/docs/ReleaseNotes.html>
Relnotes: yes
(cherry picked from commit 12cd91cf4c6b96a24427c0de5374916f2808d263)
Diffstat (limited to 'contrib/compiler-rt')
353 files changed, 13719 insertions, 5647 deletions
diff --git a/contrib/compiler-rt/include/sanitizer/common_interface_defs.h b/contrib/compiler-rt/include/sanitizer/common_interface_defs.h index f9f9302..4a1de96 100644 --- a/contrib/compiler-rt/include/sanitizer/common_interface_defs.h +++ b/contrib/compiler-rt/include/sanitizer/common_interface_defs.h @@ -158,8 +158,10 @@ extern "C" { // Prints stack traces for all live heap allocations ordered by total // allocation size until `top_percent` of total live heap is shown. // `top_percent` should be between 1 and 100. + // At most `max_number_of_contexts` contexts (stack traces) is printed. // Experimental feature currently available only with asan on Linux/x86_64. - void __sanitizer_print_memory_profile(size_t top_percent); + void __sanitizer_print_memory_profile(size_t top_percent, + size_t max_number_of_contexts); // Fiber annotation interface. // Before switching to a different stack, one must call diff --git a/contrib/compiler-rt/include/sanitizer/coverage_interface.h b/contrib/compiler-rt/include/sanitizer/coverage_interface.h index b44c5ac..637379d 100644 --- a/contrib/compiler-rt/include/sanitizer/coverage_interface.h +++ b/contrib/compiler-rt/include/sanitizer/coverage_interface.h @@ -19,8 +19,6 @@ extern "C" { #endif - // Initialize coverage. - void __sanitizer_cov_init(); // Record and dump coverage info. void __sanitizer_cov_dump(); @@ -28,42 +26,6 @@ extern "C" { // .sancov files. void __sanitizer_dump_coverage(const uintptr_t *pcs, uintptr_t len); - // Open <name>.sancov.packed in the coverage directory and return the file - // descriptor. Returns -1 on failure, or if coverage dumping is disabled. - // This is intended for use by sandboxing code. - intptr_t __sanitizer_maybe_open_cov_file(const char *name); - // Get the number of unique covered blocks (or edges). - // This can be useful for coverage-directed in-process fuzzers. - uintptr_t __sanitizer_get_total_unique_coverage(); - // Get the number of unique indirect caller-callee pairs. - uintptr_t __sanitizer_get_total_unique_caller_callee_pairs(); - - // Reset the basic-block (edge) coverage to the initial state. - // Useful for in-process fuzzing to start collecting coverage from scratch. - // Experimental, will likely not work for multi-threaded process. - void __sanitizer_reset_coverage(); - // Set *data to the array of covered PCs and return the size of that array. - // Some of the entries in *data will be zero. - uintptr_t __sanitizer_get_coverage_guards(uintptr_t **data); - - // The coverage instrumentation may optionally provide imprecise counters. - // Rather than exposing the counter values to the user we instead map - // the counters to a bitset. - // Every counter is associated with 8 bits in the bitset. - // We define 8 value ranges: 1, 2, 3, 4-7, 8-15, 16-31, 32-127, 128+ - // The i-th bit is set to 1 if the counter value is in the i-th range. - // This counter-based coverage implementation is *not* thread-safe. - - // Returns the number of registered coverage counters. - uintptr_t __sanitizer_get_number_of_counters(); - // Updates the counter 'bitset', clears the counters and returns the number of - // new bits in 'bitset'. - // If 'bitset' is nullptr, only clears the counters. - // Otherwise 'bitset' should be at least - // __sanitizer_get_number_of_counters bytes long and 8-aligned. - uintptr_t - __sanitizer_update_counter_bitset_and_clear_counters(uint8_t *bitset); - #ifdef __cplusplus } // extern "C" #endif diff --git a/contrib/compiler-rt/include/sanitizer/tsan_interface.h b/contrib/compiler-rt/include/sanitizer/tsan_interface.h new file mode 100644 index 0000000..5ea09ab --- /dev/null +++ b/contrib/compiler-rt/include/sanitizer/tsan_interface.h @@ -0,0 +1,138 @@ +//===-- tsan_interface.h ----------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of ThreadSanitizer (TSan), a race detector. +// +// Public interface header for TSan. +//===----------------------------------------------------------------------===// +#ifndef SANITIZER_TSAN_INTERFACE_H +#define SANITIZER_TSAN_INTERFACE_H + +#include <sanitizer/common_interface_defs.h> + +#ifdef __cplusplus +extern "C" { +#endif + +// __tsan_release establishes a happens-before relation with a preceding +// __tsan_acquire on the same address. +void __tsan_acquire(void *addr); +void __tsan_release(void *addr); + +// Annotations for custom mutexes. +// The annotations allow to get better reports (with sets of locked mutexes), +// detect more types of bugs (e.g. mutex misuses, races between lock/unlock and +// destruction and potential deadlocks) and improve precision and performance +// (by ignoring individual atomic operations in mutex code). However, the +// downside is that annotated mutex code itself is not checked for correctness. + +// Mutex creation flags are passed to __tsan_mutex_create annotation. +// If mutex has no constructor and __tsan_mutex_create is not called, +// the flags may be passed to __tsan_mutex_pre_lock/__tsan_mutex_post_lock +// annotations. + +// Mutex has static storage duration and no-op constructor and destructor. +// This effectively makes tsan ignore destroy annotation. +const unsigned __tsan_mutex_linker_init = 1 << 0; +// Mutex is write reentrant. +const unsigned __tsan_mutex_write_reentrant = 1 << 1; +// Mutex is read reentrant. +const unsigned __tsan_mutex_read_reentrant = 1 << 2; + +// Mutex operation flags: + +// Denotes read lock operation. +const unsigned __tsan_mutex_read_lock = 1 << 3; +// Denotes try lock operation. +const unsigned __tsan_mutex_try_lock = 1 << 4; +// Denotes that a try lock operation has failed to acquire the mutex. +const unsigned __tsan_mutex_try_lock_failed = 1 << 5; +// Denotes that the lock operation acquires multiple recursion levels. +// Number of levels is passed in recursion parameter. +// This is useful for annotation of e.g. Java builtin monitors, +// for which wait operation releases all recursive acquisitions of the mutex. +const unsigned __tsan_mutex_recursive_lock = 1 << 6; +// Denotes that the unlock operation releases all recursion levels. +// Number of released levels is returned and later must be passed to +// the corresponding __tsan_mutex_post_lock annotation. +const unsigned __tsan_mutex_recursive_unlock = 1 << 7; + +// Annotate creation of a mutex. +// Supported flags: mutex creation flags. +void __tsan_mutex_create(void *addr, unsigned flags); + +// Annotate destruction of a mutex. +// Supported flags: +// - __tsan_mutex_linker_init +void __tsan_mutex_destroy(void *addr, unsigned flags); + +// Annotate start of lock operation. +// Supported flags: +// - __tsan_mutex_read_lock +// - __tsan_mutex_try_lock +// - all mutex creation flags +void __tsan_mutex_pre_lock(void *addr, unsigned flags); + +// Annotate end of lock operation. +// Supported flags: +// - __tsan_mutex_read_lock (must match __tsan_mutex_pre_lock) +// - __tsan_mutex_try_lock (must match __tsan_mutex_pre_lock) +// - __tsan_mutex_try_lock_failed +// - __tsan_mutex_recursive_lock +// - all mutex creation flags +void __tsan_mutex_post_lock(void *addr, unsigned flags, int recursion); + +// Annotate start of unlock operation. +// Supported flags: +// - __tsan_mutex_read_lock +// - __tsan_mutex_recursive_unlock +int __tsan_mutex_pre_unlock(void *addr, unsigned flags); + +// Annotate end of unlock operation. +// Supported flags: +// - __tsan_mutex_read_lock (must match __tsan_mutex_pre_unlock) +void __tsan_mutex_post_unlock(void *addr, unsigned flags); + +// Annotate start/end of notify/signal/broadcast operation. +// Supported flags: none. +void __tsan_mutex_pre_signal(void *addr, unsigned flags); +void __tsan_mutex_post_signal(void *addr, unsigned flags); + +// Annotate start/end of a region of code where lock/unlock/signal operation +// diverts to do something else unrelated to the mutex. This can be used to +// annotate, for example, calls into cooperative scheduler or contention +// profiling code. +// These annotations must be called only from within +// __tsan_mutex_pre/post_lock, __tsan_mutex_pre/post_unlock, +// __tsan_mutex_pre/post_signal regions. +// Supported flags: none. +void __tsan_mutex_pre_divert(void *addr, unsigned flags); +void __tsan_mutex_post_divert(void *addr, unsigned flags); + +// External race detection API. +// Can be used by non-instrumented libraries to detect when their objects are +// being used in an unsafe manner. +// - __tsan_external_read/__tsan_external_write annotates the logical reads +// and writes of the object at the specified address. 'caller_pc' should +// be the PC of the library user, which the library can obtain with e.g. +// `__builtin_return_address(0)`. +// - __tsan_external_register_tag registers a 'tag' with the specified name, +// which is later used in read/write annotations to denote the object type +// - __tsan_external_assign_tag can optionally mark a heap object with a tag +void *__tsan_external_register_tag(const char *object_type); +void __tsan_external_register_header(void *tag, const char *header); +void __tsan_external_assign_tag(void *addr, void *tag); +void __tsan_external_read(void *addr, void *caller_pc, void *tag); +void __tsan_external_write(void *addr, void *caller_pc, void *tag); + +#ifdef __cplusplus +} // extern "C" +#endif + +#endif // SANITIZER_TSAN_INTERFACE_H diff --git a/contrib/compiler-rt/include/xray/xray_interface.h b/contrib/compiler-rt/include/xray/xray_interface.h index 9e712b1..5646134 100644 --- a/contrib/compiler-rt/include/xray/xray_interface.h +++ b/contrib/compiler-rt/include/xray/xray_interface.h @@ -1,4 +1,4 @@ -//===-- xray_interface.h ----------------------------------------*- C++ -*-===// +//===- xray_interface.h -----------------------------------------*- C++ -*-===// // // The LLVM Compiler Infrastructure // @@ -11,41 +11,69 @@ // // APIs for controlling XRay functionality explicitly. //===----------------------------------------------------------------------===// + #ifndef XRAY_XRAY_INTERFACE_H #define XRAY_XRAY_INTERFACE_H +#include <cstddef> #include <cstdint> extern "C" { -enum XRayEntryType { ENTRY = 0, EXIT = 1, TAIL = 2 }; +/// Synchronize this with AsmPrinter::SledKind in LLVM. +enum XRayEntryType { + ENTRY = 0, + EXIT = 1, + TAIL = 2, + LOG_ARGS_ENTRY = 3, + CUSTOM_EVENT = 4, +}; -// Provide a function to invoke for when instrumentation points are hit. This is -// a user-visible control surface that overrides the default implementation. The -// function provided should take the following arguments: -// -// - function id: an identifier that indicates the id of a function; this id -// is generated by xray; the mapping between the function id -// and the actual function pointer is available through -// __xray_table. -// - entry type: identifies what kind of instrumentation point was encountered -// (function entry, function exit, etc.). See the enum -// XRayEntryType for more details. -// -// The user handler must handle correctly spurious calls after this handler is -// removed or replaced with another handler, because it would be too costly for -// XRay runtime to avoid spurious calls. -// To prevent circular calling, the handler function itself and all its -// direct&indirect callees must not be instrumented with XRay, which can be -// achieved by marking them all with: __attribute__((xray_never_instrument)) -// -// Returns 1 on success, 0 on error. +/// Provide a function to invoke for when instrumentation points are hit. This +/// is a user-visible control surface that overrides the default implementation. +/// The function provided should take the following arguments: +/// +/// - function id: an identifier that indicates the id of a function; this id +/// is generated by xray; the mapping between the function id +/// and the actual function pointer is available through +/// __xray_table. +/// - entry type: identifies what kind of instrumentation point was +/// encountered (function entry, function exit, etc.). See the +/// enum XRayEntryType for more details. +/// +/// The user handler must handle correctly spurious calls after this handler is +/// removed or replaced with another handler, because it would be too costly for +/// XRay runtime to avoid spurious calls. +/// To prevent circular calling, the handler function itself and all its +/// direct&indirect callees must not be instrumented with XRay, which can be +/// achieved by marking them all with: __attribute__((xray_never_instrument)) +/// +/// Returns 1 on success, 0 on error. extern int __xray_set_handler(void (*entry)(int32_t, XRayEntryType)); -// This removes whatever the currently provided handler is. Returns 1 on -// success, 0 on error. +/// This removes whatever the currently provided handler is. Returns 1 on +/// success, 0 on error. extern int __xray_remove_handler(); +/// Use XRay to log the first argument of each (instrumented) function call. +/// When this function exits, all threads will have observed the effect and +/// start logging their subsequent affected function calls (if patched). +/// +/// Returns 1 on success, 0 on error. +extern int __xray_set_handler_arg1(void (*entry)(int32_t, XRayEntryType, + uint64_t)); + +/// Disables the XRay handler used to log first arguments of function calls. +/// Returns 1 on success, 0 on error. +extern int __xray_remove_handler_arg1(); + +/// Provide a function to invoke when XRay encounters a custom event. +extern int __xray_set_customevent_handler(void (*entry)(void*, std::size_t)); + +/// This removes whatever the currently provided custom event handler is. +/// Returns 1 on success, 0 on error. +extern int __xray_remove_customevent_handler(); + enum XRayPatchingStatus { NOT_INITIALIZED = 0, SUCCESS = 1, @@ -53,13 +81,31 @@ enum XRayPatchingStatus { FAILED = 3, }; -// This tells XRay to patch the instrumentation points. See XRayPatchingStatus -// for possible result values. +/// This tells XRay to patch the instrumentation points. See XRayPatchingStatus +/// for possible result values. extern XRayPatchingStatus __xray_patch(); -// Reverses the effect of __xray_patch(). See XRayPatchingStatus for possible -// result values. +/// Reverses the effect of __xray_patch(). See XRayPatchingStatus for possible +/// result values. extern XRayPatchingStatus __xray_unpatch(); -} -#endif +/// This patches a specific function id. See XRayPatchingStatus for possible +/// result values. +extern XRayPatchingStatus __xray_patch_function(int32_t FuncId); + +/// This unpatches a specific function id. See XRayPatchingStatus for possible +/// result values. +extern XRayPatchingStatus __xray_unpatch_function(int32_t FuncId); + +/// This function returns the address of the function provided a valid function +/// id. We return 0 if we encounter any error, even if 0 may be a valid function +/// address. +extern uintptr_t __xray_function_address(int32_t FuncId); + +/// This function returns the maximum valid function id. Returns 0 if we +/// encounter errors (when there are no instrumented functions, etc.). +extern size_t __xray_max_function_id(); + +} // end extern "C" + +#endif // XRAY_XRAY_INTERFACE_H diff --git a/contrib/compiler-rt/include/xray/xray_log_interface.h b/contrib/compiler-rt/include/xray/xray_log_interface.h new file mode 100644 index 0000000..cdb2009 --- /dev/null +++ b/contrib/compiler-rt/include/xray/xray_log_interface.h @@ -0,0 +1,231 @@ +//===-- xray_log_interface.h ----------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a function call tracing system. +// +// APIs for installing a new logging implementation. +// +//===----------------------------------------------------------------------===// +/// +/// XRay allows users to implement their own logging handlers and install them +/// to replace the default runtime-controllable implementation that comes with +/// compiler-rt/xray. The "flight data recorder" (FDR) mode implementation uses +/// this API to install itself in an XRay-enabled binary. See +/// compiler-rt/lib/xray_fdr_logging.{h,cc} for details of that implementation. +/// +/// The high-level usage pattern for these APIs look like the following: +/// +/// // Before we try initializing the log implementation, we must set it as +/// // the log implementation. We provide the function pointers that define +/// // the various initialization, finalization, and other pluggable hooks +/// // that we need. +/// __xray_set_log_impl({...}); +/// +/// // Once that's done, we can now initialize the implementation. Each +/// // implementation has a chance to let users customize the implementation +/// // with a struct that their implementation supports. Roughly this might +/// // look like: +/// MyImplementationOptions opts; +/// opts.enable_feature = true; +/// ... +/// auto init_status = __xray_log_init( +/// BufferSize, MaxBuffers, &opts, sizeof opts); +/// if (init_status != XRayLogInitStatus::XRAY_LOG_INITIALIZED) { +/// // deal with the error here, if there is one. +/// } +/// +/// // When the log implementation has had the chance to initialize, we can +/// // now patch the sleds. +/// auto patch_status = __xray_patch(); +/// if (patch_status != XRayPatchingStatus::SUCCESS) { +/// // deal with the error here, if it is an error. +/// } +/// +/// // If we want to stop the implementation, we can then finalize it (before +/// // optionally flushing the log). +/// auto fin_status = __xray_log_finalize(); +/// if (fin_status != XRayLogInitStatus::XRAY_LOG_FINALIZED) { +/// // deal with the error here, if it is an error. +/// } +/// +/// // We can optionally wait before flushing the log to give other threads a +/// // chance to see that the implementation is already finalized. Also, at +/// // this point we can optionally unpatch the sleds to reduce overheads at +/// // runtime. +/// auto unpatch_status = __xray_unpatch(); +/// if (unpatch_status != XRayPatchingStatus::SUCCESS) { +// // deal with the error here, if it is an error. +// } +/// +/// // If there are logs or data to be flushed somewhere, we can do so only +/// // after we've finalized the log. Some implementations may not actually +/// // have anything to log (it might keep the data in memory, or periodically +/// // be logging the data anyway). +/// auto flush_status = __xray_log_flushLog(); +/// if (flush_status != XRayLogFlushStatus::XRAY_LOG_FLUSHED) { +/// // deal with the error here, if it is an error. +/// } +/// +/// +/// NOTE: Before calling __xray_patch() again, consider re-initializing the +/// implementation first. Some implementations might stay in an "off" state when +/// they are finalized, while some might be in an invalid/unknown state. +/// +#ifndef XRAY_XRAY_LOG_INTERFACE_H +#define XRAY_XRAY_LOG_INTERFACE_H + +#include "xray/xray_interface.h" +#include <stddef.h> + +extern "C" { + +/// This enum defines the valid states in which the logging implementation can +/// be at. +enum XRayLogInitStatus { + /// The default state is uninitialized, and in case there were errors in the + /// initialization, the implementation MUST return XRAY_LOG_UNINITIALIZED. + XRAY_LOG_UNINITIALIZED = 0, + + /// Some implementations support multi-stage init (or asynchronous init), and + /// may return XRAY_LOG_INITIALIZING to signal callers of the API that + /// there's an ongoing initialization routine running. This allows + /// implementations to support concurrent threads attempting to initialize, + /// while only signalling success in one. + XRAY_LOG_INITIALIZING = 1, + + /// When an implementation is done initializing, it MUST return + /// XRAY_LOG_INITIALIZED. When users call `__xray_patch()`, they are + /// guaranteed that the implementation installed with + /// `__xray_set_log_impl(...)` has been initialized. + XRAY_LOG_INITIALIZED = 2, + + /// Some implementations might support multi-stage finalization (or + /// asynchronous finalization), and may return XRAY_LOG_FINALIZING to signal + /// callers of the API that there's an ongoing finalization routine running. + /// This allows implementations to support concurrent threads attempting to + /// finalize, while only signalling success/completion in one. + XRAY_LOG_FINALIZING = 3, + + /// When an implementation is done finalizing, it MUST return + /// XRAY_LOG_FINALIZED. It is up to the implementation to determine what the + /// semantics of a finalized implementation is. Some implementations might + /// allow re-initialization once the log is finalized, while some might always + /// be on (and that finalization is a no-op). + XRAY_LOG_FINALIZED = 4, +}; + +/// This enum allows an implementation to signal log flushing operations via +/// `__xray_log_flushLog()`, and the state of flushing the log. +enum XRayLogFlushStatus { + XRAY_LOG_NOT_FLUSHING = 0, + XRAY_LOG_FLUSHING = 1, + XRAY_LOG_FLUSHED = 2, +}; + +/// A valid XRay logging implementation MUST provide all of the function +/// pointers in XRayLogImpl when being installed through `__xray_set_log_impl`. +/// To be precise, ALL the functions pointers MUST NOT be nullptr. +struct XRayLogImpl { + /// The log initialization routine provided by the implementation, always + /// provided with the following parameters: + /// + /// - buffer size + /// - maximum number of buffers + /// - a pointer to an argument struct that the implementation MUST handle + /// - the size of the argument struct + /// + /// See XRayLogInitStatus for details on what the implementation MUST return + /// when called. + /// + /// If the implementation needs to install handlers aside from the 0-argument + /// function call handler, it MUST do so in this initialization handler. + /// + /// See xray_interface.h for available handler installation routines. + XRayLogInitStatus (*log_init)(size_t, size_t, void *, size_t); + + /// The log finalization routine provided by the implementation. + /// + /// See XRayLogInitStatus for details on what the implementation MUST return + /// when called. + XRayLogInitStatus (*log_finalize)(); + + /// The 0-argument function call handler. XRay logging implementations MUST + /// always have a handler for function entry and exit events. In case the + /// implementation wants to support arg1 (or other future extensions to XRay + /// logging) those MUST be installed by the installed 'log_init' handler. + void (*handle_arg0)(int32_t, XRayEntryType); + + /// The log implementation provided routine for when __xray_log_flushLog() is + /// called. + /// + /// See XRayLogFlushStatus for details on what the implementation MUST return + /// when called. + XRayLogFlushStatus (*flush_log)(); +}; + +/// This function installs a new logging implementation that XRay will use. In +/// case there are any nullptr members in Impl, XRay will *uninstall any +/// existing implementations*. It does NOT patch the instrumentation sleds. +/// +/// NOTE: This function does NOT attempt to finalize the currently installed +/// implementation. Use with caution. +/// +/// It is guaranteed safe to call this function in the following states: +/// +/// - When the implementation is UNINITIALIZED. +/// - When the implementation is FINALIZED. +/// - When there is no current implementation installed. +/// +/// It is logging implementation defined what happens when this function is +/// called while in any other states. +void __xray_set_log_impl(XRayLogImpl Impl); + +/// This function removes the currently installed implementation. It will also +/// uninstall any handlers that have been previously installed. It does NOT +/// unpatch the instrumentation sleds. +/// +/// NOTE: This function does NOT attempt to finalize the currently installed +/// implementation. Use with caution. +/// +/// It is guaranteed safe to call this function in the following states: +/// +/// - When the implementation is UNINITIALIZED. +/// - When the implementation is FINALIZED. +/// - When there is no current implementation installed. +/// +/// It is logging implementation defined what happens when this function is +/// called while in any other states. +void __xray_remove_log_impl(); + +/// Invokes the installed implementation initialization routine. See +/// XRayLogInitStatus for what the return values mean. +XRayLogInitStatus __xray_log_init(size_t BufferSize, size_t MaxBuffers, + void *Args, size_t ArgsSize); + +/// Invokes the installed implementation finalization routine. See +/// XRayLogInitStatus for what the return values mean. +XRayLogInitStatus __xray_log_finalize(); + +/// Invokes the install implementation log flushing routine. See +/// XRayLogFlushStatus for what the return values mean. +XRayLogFlushStatus __xray_log_flushLog(); + +} // extern "C" + +namespace __xray { + +// Options used by the LLVM XRay FDR implementation. +struct FDRLoggingOptions { + bool ReportErrors = false; + int Fd = -1; +}; + +} // namespace __xray + +#endif // XRAY_XRAY_LOG_INTERFACE_H diff --git a/contrib/compiler-rt/include/xray/xray_records.h b/contrib/compiler-rt/include/xray/xray_records.h index 34c236b..feb8d22 100644 --- a/contrib/compiler-rt/include/xray/xray_records.h +++ b/contrib/compiler-rt/include/xray/xray_records.h @@ -21,8 +21,17 @@ namespace __xray { enum FileTypes { NAIVE_LOG = 0, + FDR_LOG = 1, }; +// FDR mode use of the union field in the XRayFileHeader. +struct alignas(16) FdrAdditionalHeaderData { + uint64_t ThreadBufferSize; +}; + +static_assert(sizeof(FdrAdditionalHeaderData) == 16, + "FdrAdditionalHeaderData != 16 bytes"); + // This data structure is used to describe the contents of the file. We use this // for versioning the supported XRay file formats. struct alignas(32) XRayFileHeader { @@ -40,6 +49,16 @@ struct alignas(32) XRayFileHeader { // The frequency by which TSC increases per-second. alignas(8) uint64_t CycleFrequency = 0; + + union { + char FreeForm[16]; + // The current civiltime timestamp, as retrived from 'clock_gettime'. This + // allows readers of the file to determine when the file was created or + // written down. + struct timespec TS; + + struct FdrAdditionalHeaderData FdrData; + }; } __attribute__((packed)); static_assert(sizeof(XRayFileHeader) == 32, "XRayFileHeader != 32 bytes"); diff --git a/contrib/compiler-rt/lib/asan/asan.syms.extra b/contrib/compiler-rt/lib/asan/asan.syms.extra index 007aafe..f8e9b3a 100644 --- a/contrib/compiler-rt/lib/asan/asan.syms.extra +++ b/contrib/compiler-rt/lib/asan/asan.syms.extra @@ -1,3 +1,4 @@ __asan_* __lsan_* __ubsan_* +__sancov_* diff --git a/contrib/compiler-rt/lib/asan/asan_activation.cc b/contrib/compiler-rt/lib/asan/asan_activation.cc index 7e4e604..66eba9c 100644 --- a/contrib/compiler-rt/lib/asan/asan_activation.cc +++ b/contrib/compiler-rt/lib/asan/asan_activation.cc @@ -106,7 +106,6 @@ void AsanDeactivate() { // Deactivate the runtime. SetCanPoisonMemory(false); SetMallocContextSize(1); - ReInitializeCoverage(false, nullptr); AllocatorOptions disabled = asan_deactivated_flags.allocator_options; disabled.quarantine_size_mb = 0; @@ -130,8 +129,6 @@ void AsanActivate() { SetCanPoisonMemory(asan_deactivated_flags.poison_heap); SetMallocContextSize(asan_deactivated_flags.malloc_context_size); - ReInitializeCoverage(asan_deactivated_flags.coverage, - asan_deactivated_flags.coverage_dir); ReInitializeAllocator(asan_deactivated_flags.allocator_options); asan_is_deactivated = false; diff --git a/contrib/compiler-rt/lib/asan/asan_allocator.cc b/contrib/compiler-rt/lib/asan/asan_allocator.cc index ee9b1a6..92963dd 100644 --- a/contrib/compiler-rt/lib/asan/asan_allocator.cc +++ b/contrib/compiler-rt/lib/asan/asan_allocator.cc @@ -21,7 +21,9 @@ #include "asan_report.h" #include "asan_stack.h" #include "asan_thread.h" +#include "sanitizer_common/sanitizer_allocator_checks.h" #include "sanitizer_common/sanitizer_allocator_interface.h" +#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_flags.h" #include "sanitizer_common/sanitizer_internal_defs.h" #include "sanitizer_common/sanitizer_list.h" @@ -160,7 +162,11 @@ struct QuarantineCallback { } void *Allocate(uptr size) { - return get_allocator().Allocate(cache_, size, 1, false); + void *res = get_allocator().Allocate(cache_, size, 1); + // TODO(alekseys): Consider making quarantine OOM-friendly. + if (UNLIKELY(!res)) + return DieOnFailure::OnOOM(); + return res; } void Deallocate(void *p) { @@ -235,6 +241,8 @@ struct Allocator { AllocatorCache fallback_allocator_cache; QuarantineCache fallback_quarantine_cache; + atomic_uint8_t rss_limit_exceeded; + // ------------------- Options -------------------------- atomic_uint16_t min_redzone; atomic_uint16_t max_redzone; @@ -264,10 +272,19 @@ struct Allocator { } void Initialize(const AllocatorOptions &options) { - allocator.Init(options.may_return_null, options.release_to_os_interval_ms); + SetAllocatorMayReturnNull(options.may_return_null); + allocator.Init(options.release_to_os_interval_ms); SharedInitCode(options); } + bool RssLimitExceeded() { + return atomic_load(&rss_limit_exceeded, memory_order_relaxed); + } + + void SetRssLimitExceeded(bool limit_exceeded) { + atomic_store(&rss_limit_exceeded, limit_exceeded, memory_order_relaxed); + } + void RePoisonChunk(uptr chunk) { // This could be a user-facing chunk (with redzones), or some internal // housekeeping chunk, like TransferBatch. Start by assuming the former. @@ -292,7 +309,7 @@ struct Allocator { } void ReInitialize(const AllocatorOptions &options) { - allocator.SetMayReturnNull(options.may_return_null); + SetAllocatorMayReturnNull(options.may_return_null); allocator.SetReleaseToOSIntervalMs(options.release_to_os_interval_ms); SharedInitCode(options); @@ -313,7 +330,7 @@ struct Allocator { options->thread_local_quarantine_size_kb = quarantine.GetCacheSize() >> 10; options->min_redzone = atomic_load(&min_redzone, memory_order_acquire); options->max_redzone = atomic_load(&max_redzone, memory_order_acquire); - options->may_return_null = allocator.MayReturnNull(); + options->may_return_null = AllocatorMayReturnNull(); options->alloc_dealloc_mismatch = atomic_load(&alloc_dealloc_mismatch, memory_order_acquire); options->release_to_os_interval_ms = allocator.ReleaseToOSIntervalMs(); @@ -363,6 +380,8 @@ struct Allocator { AllocType alloc_type, bool can_fill) { if (UNLIKELY(!asan_inited)) AsanInitFromRtl(); + if (RssLimitExceeded()) + return AsanAllocator::FailureHandler::OnOOM(); Flags &fl = *flags(); CHECK(stack); const uptr min_alignment = SHADOW_GRANULARITY; @@ -395,24 +414,21 @@ struct Allocator { if (size > kMaxAllowedMallocSize || needed_size > kMaxAllowedMallocSize) { Report("WARNING: AddressSanitizer failed to allocate 0x%zx bytes\n", (void*)size); - return allocator.ReturnNullOrDieOnBadRequest(); + return AsanAllocator::FailureHandler::OnBadRequest(); } AsanThread *t = GetCurrentThread(); void *allocated; - bool check_rss_limit = true; if (t) { AllocatorCache *cache = GetAllocatorCache(&t->malloc_storage()); - allocated = - allocator.Allocate(cache, needed_size, 8, false, check_rss_limit); + allocated = allocator.Allocate(cache, needed_size, 8); } else { SpinMutexLock l(&fallback_mutex); AllocatorCache *cache = &fallback_allocator_cache; - allocated = - allocator.Allocate(cache, needed_size, 8, false, check_rss_limit); + allocated = allocator.Allocate(cache, needed_size, 8); } - - if (!allocated) return allocator.ReturnNullOrDieOnOOM(); + if (!allocated) + return nullptr; if (*(u8 *)MEM_TO_SHADOW((uptr)allocated) == 0 && CanPoisonMemory()) { // Heap poisoning is enabled, but the allocator provides an unpoisoned @@ -514,8 +530,7 @@ struct Allocator { // Expects the chunk to already be marked as quarantined by using // AtomicallySetQuarantineFlagIfAllocated. - void QuarantineChunk(AsanChunk *m, void *ptr, BufferedStackTrace *stack, - AllocType alloc_type) { + void QuarantineChunk(AsanChunk *m, void *ptr, BufferedStackTrace *stack) { CHECK_EQ(m->chunk_state, CHUNK_QUARANTINE); CHECK_GE(m->alloc_tid, 0); if (SANITIZER_WORDSIZE == 64) // On 32-bits this resides in user area. @@ -523,6 +538,18 @@ struct Allocator { AsanThread *t = GetCurrentThread(); m->free_tid = t ? t->tid() : 0; m->free_context_id = StackDepotPut(*stack); + + Flags &fl = *flags(); + if (fl.max_free_fill_size > 0) { + // We have to skip the chunk header, it contains free_context_id. + uptr scribble_start = (uptr)m + kChunkHeaderSize + kChunkHeader2Size; + if (m->UsedSize() >= kChunkHeader2Size) { // Skip Header2 in user area. + uptr size_to_fill = m->UsedSize() - kChunkHeader2Size; + size_to_fill = Min(size_to_fill, (uptr)fl.max_free_fill_size); + REAL(memset)((void *)scribble_start, fl.free_fill_byte, size_to_fill); + } + } + // Poison the region. PoisonShadow(m->Beg(), RoundUpTo(m->UsedSize(), SHADOW_GRANULARITY), @@ -554,7 +581,17 @@ struct Allocator { uptr chunk_beg = p - kChunkHeaderSize; AsanChunk *m = reinterpret_cast<AsanChunk *>(chunk_beg); + // On Windows, uninstrumented DLLs may allocate memory before ASan hooks + // malloc. Don't report an invalid free in this case. + if (SANITIZER_WINDOWS && + !get_allocator().PointerIsMine(ptr)) { + if (!IsSystemHeapAddress(p)) + ReportFreeNotMalloced(p, stack); + return; + } + ASAN_FREE_HOOK(ptr); + // Must mark the chunk as quarantined before any changes to its metadata. // Do not quarantine given chunk if we failed to set CHUNK_QUARANTINE flag. if (!AtomicallySetQuarantineFlagIfAllocated(m, ptr, stack)) return; @@ -571,7 +608,7 @@ struct Allocator { ReportNewDeleteSizeMismatch(p, delete_size, stack); } - QuarantineChunk(m, ptr, stack, alloc_type); + QuarantineChunk(m, ptr, stack); } void *Reallocate(void *old_ptr, uptr new_size, BufferedStackTrace *stack) { @@ -600,8 +637,8 @@ struct Allocator { } void *Calloc(uptr nmemb, uptr size, BufferedStackTrace *stack) { - if (CallocShouldReturnNullDueToOverflow(size, nmemb)) - return allocator.ReturnNullOrDieOnBadRequest(); + if (CheckForCallocOverflow(size, nmemb)) + return AsanAllocator::FailureHandler::OnBadRequest(); void *ptr = Allocate(nmemb * size, 8, stack, FROM_MALLOC, false); // If the memory comes from the secondary allocator no need to clear it // as it comes directly from mmap. @@ -764,11 +801,6 @@ void PrintInternalAllocatorStats() { instance.PrintStats(); } -void *asan_memalign(uptr alignment, uptr size, BufferedStackTrace *stack, - AllocType alloc_type) { - return instance.Allocate(size, alignment, stack, alloc_type, true); -} - void asan_free(void *ptr, BufferedStackTrace *stack, AllocType alloc_type) { instance.Deallocate(ptr, 0, stack, alloc_type); } @@ -779,40 +811,59 @@ void asan_sized_free(void *ptr, uptr size, BufferedStackTrace *stack, } void *asan_malloc(uptr size, BufferedStackTrace *stack) { - return instance.Allocate(size, 8, stack, FROM_MALLOC, true); + return SetErrnoOnNull(instance.Allocate(size, 8, stack, FROM_MALLOC, true)); } void *asan_calloc(uptr nmemb, uptr size, BufferedStackTrace *stack) { - return instance.Calloc(nmemb, size, stack); + return SetErrnoOnNull(instance.Calloc(nmemb, size, stack)); } void *asan_realloc(void *p, uptr size, BufferedStackTrace *stack) { if (!p) - return instance.Allocate(size, 8, stack, FROM_MALLOC, true); + return SetErrnoOnNull(instance.Allocate(size, 8, stack, FROM_MALLOC, true)); if (size == 0) { - instance.Deallocate(p, 0, stack, FROM_MALLOC); - return nullptr; + if (flags()->allocator_frees_and_returns_null_on_realloc_zero) { + instance.Deallocate(p, 0, stack, FROM_MALLOC); + return nullptr; + } + // Allocate a size of 1 if we shouldn't free() on Realloc to 0 + size = 1; } - return instance.Reallocate(p, size, stack); + return SetErrnoOnNull(instance.Reallocate(p, size, stack)); } void *asan_valloc(uptr size, BufferedStackTrace *stack) { - return instance.Allocate(size, GetPageSizeCached(), stack, FROM_MALLOC, true); + return SetErrnoOnNull( + instance.Allocate(size, GetPageSizeCached(), stack, FROM_MALLOC, true)); } void *asan_pvalloc(uptr size, BufferedStackTrace *stack) { uptr PageSize = GetPageSizeCached(); - size = RoundUpTo(size, PageSize); - if (size == 0) { - // pvalloc(0) should allocate one page. - size = PageSize; + // pvalloc(0) should allocate one page. + size = size ? RoundUpTo(size, PageSize) : PageSize; + return SetErrnoOnNull( + instance.Allocate(size, PageSize, stack, FROM_MALLOC, true)); +} + +void *asan_memalign(uptr alignment, uptr size, BufferedStackTrace *stack, + AllocType alloc_type) { + if (UNLIKELY(!IsPowerOfTwo(alignment))) { + errno = errno_EINVAL; + return AsanAllocator::FailureHandler::OnBadRequest(); } - return instance.Allocate(size, PageSize, stack, FROM_MALLOC, true); + return SetErrnoOnNull( + instance.Allocate(size, alignment, stack, alloc_type, true)); } int asan_posix_memalign(void **memptr, uptr alignment, uptr size, BufferedStackTrace *stack) { + if (UNLIKELY(!CheckPosixMemalignAlignment(alignment))) { + AsanAllocator::FailureHandler::OnBadRequest(); + return errno_EINVAL; + } void *ptr = instance.Allocate(size, alignment, stack, FROM_MALLOC, true); + if (UNLIKELY(!ptr)) + return errno_ENOMEM; CHECK(IsAligned((uptr)ptr, alignment)); *memptr = ptr; return 0; @@ -840,8 +891,8 @@ void asan_mz_force_unlock() { instance.ForceUnlock(); } -void AsanSoftRssLimitExceededCallback(bool exceeded) { - instance.allocator.SetRssLimitIsExceeded(exceeded); +void AsanSoftRssLimitExceededCallback(bool limit_exceeded) { + instance.SetRssLimitExceeded(limit_exceeded); } } // namespace __asan @@ -958,15 +1009,13 @@ uptr __sanitizer_get_allocated_size(const void *p) { #if !SANITIZER_SUPPORTS_WEAK_HOOKS // Provide default (no-op) implementation of malloc hooks. -extern "C" { -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_malloc_hook(void *ptr, uptr size) { +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_malloc_hook, + void *ptr, uptr size) { (void)ptr; (void)size; } -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_free_hook(void *ptr) { + +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_free_hook, void *ptr) { (void)ptr; } -} // extern "C" #endif diff --git a/contrib/compiler-rt/lib/asan/asan_allocator.h b/contrib/compiler-rt/lib/asan/asan_allocator.h index ee28ecf..ad1aeb5 100644 --- a/contrib/compiler-rt/lib/asan/asan_allocator.h +++ b/contrib/compiler-rt/lib/asan/asan_allocator.h @@ -161,10 +161,17 @@ typedef FlatByteMap<kNumRegions> ByteMap; typedef TwoLevelByteMap<(kNumRegions >> 12), 1 << 12> ByteMap; # endif typedef CompactSizeClassMap SizeClassMap; -typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, 16, - SizeClassMap, kRegionSizeLog, - ByteMap, - AsanMapUnmapCallback> PrimaryAllocator; +struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = 16; + typedef __asan::SizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = __asan::kRegionSizeLog; + typedef __asan::ByteMap ByteMap; + typedef AsanMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; +}; +typedef SizeClassAllocator32<AP32> PrimaryAllocator; #endif // SANITIZER_CAN_USE_ALLOCATOR64 static const uptr kNumberOfSizeClasses = SizeClassMap::kNumClasses; diff --git a/contrib/compiler-rt/lib/asan/asan_descriptions.cc b/contrib/compiler-rt/lib/asan/asan_descriptions.cc index 0ecbe09..822a6a6 100644 --- a/contrib/compiler-rt/lib/asan/asan_descriptions.cc +++ b/contrib/compiler-rt/lib/asan/asan_descriptions.cc @@ -252,6 +252,9 @@ static void PrintAccessAndVarIntersection(const StackVarDescr &var, uptr addr, str.append("%c", var.name_pos[i]); } str.append("'"); + if (var.line > 0) { + str.append(" (line %d)", var.line); + } if (pos_descr) { Decorator d; // FIXME: we may want to also print the size of the access here, diff --git a/contrib/compiler-rt/lib/asan/asan_errors.cc b/contrib/compiler-rt/lib/asan/asan_errors.cc index c287ba1..b7a38eb 100644 --- a/contrib/compiler-rt/lib/asan/asan_errors.cc +++ b/contrib/compiler-rt/lib/asan/asan_errors.cc @@ -58,10 +58,21 @@ static void MaybeDumpRegisters(void *context) { SignalContext::DumpAllRegisters(context); } +static void MaybeReportNonExecRegion(uptr pc) { +#if SANITIZER_FREEBSD || SANITIZER_LINUX + MemoryMappingLayout proc_maps(/*cache_enabled*/ true); + MemoryMappedSegment segment; + while (proc_maps.Next(&segment)) { + if (pc >= segment.start && pc < segment.end && !segment.IsExecutable()) + Report("Hint: PC is at a non-executable region. Maybe a wild jump?\n"); + } +#endif +} + void ErrorDeadlySignal::Print() { Decorator d; Printf("%s", d.Warning()); - const char *description = DescribeSignalOrException(signo); + const char *description = __sanitizer::DescribeSignalOrException(signo); Report( "ERROR: AddressSanitizer: %s on unknown address %p (pc %p bp %p sp %p " "T%d)\n", @@ -77,6 +88,7 @@ void ErrorDeadlySignal::Print() { if (addr < GetPageSizeCached()) Report("Hint: address points to the zero page.\n"); } + MaybeReportNonExecRegion(pc); scariness.Print(); BufferedStackTrace stack; GetStackTraceWithPcBpAndContext(&stack, kStackTraceMax, pc, bp, context, diff --git a/contrib/compiler-rt/lib/asan/asan_flags.cc b/contrib/compiler-rt/lib/asan/asan_flags.cc index ad5bbff..6be0d6e 100644 --- a/contrib/compiler-rt/lib/asan/asan_flags.cc +++ b/contrib/compiler-rt/lib/asan/asan_flags.cc @@ -61,7 +61,7 @@ void InitializeFlags() { { CommonFlags cf; cf.CopyFrom(*common_flags()); - cf.detect_leaks = CAN_SANITIZE_LEAKS; + cf.detect_leaks = cf.detect_leaks && CAN_SANITIZE_LEAKS; cf.external_symbolizer_path = GetEnv("ASAN_SYMBOLIZER_PATH"); cf.malloc_context_size = kDefaultMallocContextSize; cf.intercept_tls_get_addr = true; @@ -95,6 +95,18 @@ void InitializeFlags() { RegisterCommonFlags(&ubsan_parser); #endif + if (SANITIZER_MAC) { + // Support macOS MallocScribble and MallocPreScribble: + // <https://developer.apple.com/library/content/documentation/Performance/ + // Conceptual/ManagingMemory/Articles/MallocDebug.html> + if (GetEnv("MallocScribble")) { + f->max_free_fill_size = 0x1000; + } + if (GetEnv("MallocPreScribble")) { + f->malloc_fill_byte = 0xaa; + } + } + // Override from ASan compile definition. const char *asan_compile_def = MaybeUseAsanDefaultOptionsCompileDefinition(); asan_parser.ParseString(asan_compile_def); @@ -182,13 +194,14 @@ void InitializeFlags() { Report("WARNING: strchr* interceptors are enabled even though " "replace_str=0. Use intercept_strchr=0 to disable them."); } + if (!f->replace_str && common_flags()->intercept_strndup) { + Report("WARNING: strndup* interceptors are enabled even though " + "replace_str=0. Use intercept_strndup=0 to disable them."); + } } } // namespace __asan -#if !SANITIZER_SUPPORTS_WEAK_HOOKS -extern "C" { -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -const char* __asan_default_options() { return ""; } -} // extern "C" -#endif +SANITIZER_INTERFACE_WEAK_DEF(const char*, __asan_default_options, void) { + return ""; +} diff --git a/contrib/compiler-rt/lib/asan/asan_flags.inc b/contrib/compiler-rt/lib/asan/asan_flags.inc index 4712efb..f2216c2 100644 --- a/contrib/compiler-rt/lib/asan/asan_flags.inc +++ b/contrib/compiler-rt/lib/asan/asan_flags.inc @@ -63,8 +63,14 @@ ASAN_FLAG( int, max_malloc_fill_size, 0x1000, // By default, fill only the first 4K. "ASan allocator flag. max_malloc_fill_size is the maximal amount of " "bytes that will be filled with malloc_fill_byte on malloc.") +ASAN_FLAG( + int, max_free_fill_size, 0, + "ASan allocator flag. max_free_fill_size is the maximal amount of " + "bytes that will be filled with free_fill_byte during free.") ASAN_FLAG(int, malloc_fill_byte, 0xbe, "Value used to fill the newly allocated memory.") +ASAN_FLAG(int, free_fill_byte, 0x55, + "Value used to fill deallocated memory.") ASAN_FLAG(bool, allow_user_poisoning, true, "If set, user may manually mark memory regions as poisoned or " "unpoisoned.") @@ -148,3 +154,10 @@ ASAN_FLAG(bool, halt_on_error, true, "(WARNING: USE AT YOUR OWN RISK!)") ASAN_FLAG(bool, use_odr_indicator, false, "Use special ODR indicator symbol for ODR violation detection") +ASAN_FLAG(bool, allocator_frees_and_returns_null_on_realloc_zero, true, + "realloc(p, 0) is equivalent to free(p) by default (Same as the " + "POSIX standard). If set to false, realloc(p, 0) will return a " + "pointer to an allocated space which can not be used.") +ASAN_FLAG(bool, verify_asan_link_order, true, + "Check position of ASan runtime in library list (needs to be disabled" + " when other library has to be preloaded system-wide)") diff --git a/contrib/compiler-rt/lib/asan/asan_globals.cc b/contrib/compiler-rt/lib/asan/asan_globals.cc index b723306..eebada8 100644 --- a/contrib/compiler-rt/lib/asan/asan_globals.cc +++ b/contrib/compiler-rt/lib/asan/asan_globals.cc @@ -332,6 +332,26 @@ void __asan_unregister_image_globals(uptr *flag) { *flag = 0; } +void __asan_register_elf_globals(uptr *flag, void *start, void *stop) { + if (*flag) return; + if (!start) return; + CHECK_EQ(0, ((uptr)stop - (uptr)start) % sizeof(__asan_global)); + __asan_global *globals_start = (__asan_global*)start; + __asan_global *globals_stop = (__asan_global*)stop; + __asan_register_globals(globals_start, globals_stop - globals_start); + *flag = 1; +} + +void __asan_unregister_elf_globals(uptr *flag, void *start, void *stop) { + if (!*flag) return; + if (!start) return; + CHECK_EQ(0, ((uptr)stop - (uptr)start) % sizeof(__asan_global)); + __asan_global *globals_start = (__asan_global*)start; + __asan_global *globals_stop = (__asan_global*)stop; + __asan_unregister_globals(globals_start, globals_stop - globals_start); + *flag = 0; +} + // Register an array of globals. void __asan_register_globals(__asan_global *globals, uptr n) { if (!flags()->report_globals) return; diff --git a/contrib/compiler-rt/lib/asan/asan_globals_win.cc b/contrib/compiler-rt/lib/asan/asan_globals_win.cc index 56c0d1a5..261762b 100644 --- a/contrib/compiler-rt/lib/asan/asan_globals_win.cc +++ b/contrib/compiler-rt/lib/asan/asan_globals_win.cc @@ -29,7 +29,7 @@ static void call_on_globals(void (*hook)(__asan_global *, uptr)) { __asan_global *end = &__asan_globals_end; uptr bytediff = (uptr)end - (uptr)start; if (bytediff % sizeof(__asan_global) != 0) { -#ifdef ASAN_DLL_THUNK +#ifdef SANITIZER_DLL_THUNK __debugbreak(); #else CHECK("corrupt asan global array"); diff --git a/contrib/compiler-rt/lib/asan/asan_globals_win.h b/contrib/compiler-rt/lib/asan/asan_globals_win.h deleted file mode 100644 index d4ed9c1..0000000 --- a/contrib/compiler-rt/lib/asan/asan_globals_win.h +++ /dev/null @@ -1,34 +0,0 @@ -//===-- asan_globals_win.h --------------------------------------*- C++ -*-===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// Interface to the Windows-specific global management code. Separated into a -// standalone header to allow inclusion from asan_win_dynamic_runtime_thunk, -// which defines symbols that clash with other sanitizer headers. -// -//===----------------------------------------------------------------------===// - -#ifndef ASAN_GLOBALS_WIN_H -#define ASAN_GLOBALS_WIN_H - -#if !defined(_MSC_VER) -#error "this file is Windows-only, and uses MSVC pragmas" -#endif - -#if defined(_WIN64) -#define SANITIZER_SYM_PREFIX -#else -#define SANITIZER_SYM_PREFIX "_" -#endif - -// Use this macro to force linking asan_globals_win.cc into the DSO. -#define ASAN_LINK_GLOBALS_WIN() \ - __pragma( \ - comment(linker, "/include:" SANITIZER_SYM_PREFIX "__asan_dso_reg_hook")) - -#endif // ASAN_GLOBALS_WIN_H diff --git a/contrib/compiler-rt/lib/asan/asan_interceptors.cc b/contrib/compiler-rt/lib/asan/asan_interceptors.cc index 606016d..34ca22b 100644 --- a/contrib/compiler-rt/lib/asan/asan_interceptors.cc +++ b/contrib/compiler-rt/lib/asan/asan_interceptors.cc @@ -37,12 +37,19 @@ namespace __asan { // Return true if we can quickly decide that the region is unpoisoned. +// We assume that a redzone is at least 16 bytes. static inline bool QuickCheckForUnpoisonedRegion(uptr beg, uptr size) { if (size == 0) return true; if (size <= 32) return !AddressIsPoisoned(beg) && !AddressIsPoisoned(beg + size - 1) && !AddressIsPoisoned(beg + size / 2); + if (size <= 64) + return !AddressIsPoisoned(beg) && + !AddressIsPoisoned(beg + size / 4) && + !AddressIsPoisoned(beg + size - 1) && + !AddressIsPoisoned(beg + 3 * size / 4) && + !AddressIsPoisoned(beg + size / 2); return false; } @@ -171,6 +178,10 @@ void SetThreadName(const char *name) { } int OnExit() { + if (CAN_SANITIZE_LEAKS && common_flags()->detect_leaks && + __lsan::HasReportedLeaks()) { + return common_flags()->exitcode; + } // FIXME: ask frontend whether we need to return failure. return 0; } @@ -228,13 +239,14 @@ DECLARE_REAL_AND_INTERCEPTOR(void, free, void *) // Strict init-order checking is dlopen-hostile: // https://github.com/google/sanitizers/issues/178 #define COMMON_INTERCEPTOR_ON_DLOPEN(filename, flag) \ - if (flags()->strict_init_order) { \ - StopInitOrderChecking(); \ - } + do { \ + if (flags()->strict_init_order) \ + StopInitOrderChecking(); \ + CheckNoDeepBind(filename, flag); \ + } while (false) #define COMMON_INTERCEPTOR_ON_EXIT(ctx) OnExit() -#define COMMON_INTERCEPTOR_LIBRARY_LOADED(filename, handle) \ - CoverageUpdateMapping() -#define COMMON_INTERCEPTOR_LIBRARY_UNLOADED() CoverageUpdateMapping() +#define COMMON_INTERCEPTOR_LIBRARY_LOADED(filename, handle) +#define COMMON_INTERCEPTOR_LIBRARY_UNLOADED() #define COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED (!asan_inited) #define COMMON_INTERCEPTOR_GET_TLS_RANGE(begin, end) \ if (AsanThread *t = GetCurrentThread()) { \ @@ -348,28 +360,22 @@ DEFINE_REAL_PTHREAD_FUNCTIONS #if SANITIZER_ANDROID INTERCEPTOR(void*, bsd_signal, int signum, void *handler) { - if (!IsHandledDeadlySignal(signum) || - common_flags()->allow_user_segv_handler) { + if (GetHandleSignalMode(signum) != kHandleSignalExclusive) return REAL(bsd_signal)(signum, handler); - } return 0; } #endif INTERCEPTOR(void*, signal, int signum, void *handler) { - if (!IsHandledDeadlySignal(signum) || - common_flags()->allow_user_segv_handler) { + if (GetHandleSignalMode(signum) != kHandleSignalExclusive) return REAL(signal)(signum, handler); - } return nullptr; } INTERCEPTOR(int, sigaction, int signum, const struct sigaction *act, struct sigaction *oldact) { - if (!IsHandledDeadlySignal(signum) || - common_flags()->allow_user_segv_handler) { + if (GetHandleSignalMode(signum) != kHandleSignalExclusive) return REAL(sigaction)(signum, act, oldact); - } return 0; } @@ -434,6 +440,13 @@ INTERCEPTOR(void, _longjmp, void *env, int val) { } #endif +#if ASAN_INTERCEPT___LONGJMP_CHK +INTERCEPTOR(void, __longjmp_chk, void *env, int val) { + __asan_handle_no_return(); + REAL(__longjmp_chk)(env, val); +} +#endif + #if ASAN_INTERCEPT_SIGLONGJMP INTERCEPTOR(void, siglongjmp, void *env, int val) { __asan_handle_no_return(); @@ -570,17 +583,6 @@ INTERCEPTOR(char*, __strdup, const char *s) { } #endif // ASAN_INTERCEPT___STRDUP -INTERCEPTOR(SIZE_T, wcslen, const wchar_t *s) { - void *ctx; - ASAN_INTERCEPTOR_ENTER(ctx, wcslen); - SIZE_T length = internal_wcslen(s); - if (!asan_init_is_running) { - ENSURE_ASAN_INITED(); - ASAN_READ_RANGE(ctx, s, (length + 1) * sizeof(wchar_t)); - } - return length; -} - INTERCEPTOR(char*, strncpy, char *to, const char *from, uptr size) { void *ctx; ASAN_INTERCEPTOR_ENTER(ctx, strncpy); @@ -697,9 +699,7 @@ INTERCEPTOR(int, __cxa_atexit, void (*func)(void *), void *arg, #if ASAN_INTERCEPT_FORK INTERCEPTOR(int, fork, void) { ENSURE_ASAN_INITED(); - if (common_flags()->coverage) CovBeforeFork(); int pid = REAL(fork)(); - if (common_flags()->coverage) CovAfterFork(pid); return pid; } #endif // ASAN_INTERCEPT_FORK @@ -715,7 +715,6 @@ void InitializeAsanInterceptors() { // Intercept str* functions. ASAN_INTERCEPT_FUNC(strcat); // NOLINT ASAN_INTERCEPT_FUNC(strcpy); // NOLINT - ASAN_INTERCEPT_FUNC(wcslen); ASAN_INTERCEPT_FUNC(strncat); ASAN_INTERCEPT_FUNC(strncpy); ASAN_INTERCEPT_FUNC(strdup); @@ -749,6 +748,9 @@ void InitializeAsanInterceptors() { #if ASAN_INTERCEPT__LONGJMP ASAN_INTERCEPT_FUNC(_longjmp); #endif +#if ASAN_INTERCEPT___LONGJMP_CHK + ASAN_INTERCEPT_FUNC(__longjmp_chk); +#endif #if ASAN_INTERCEPT_SIGLONGJMP ASAN_INTERCEPT_FUNC(siglongjmp); #endif diff --git a/contrib/compiler-rt/lib/asan/asan_interceptors.h b/contrib/compiler-rt/lib/asan/asan_interceptors.h index d747c31..93fca4f 100644 --- a/contrib/compiler-rt/lib/asan/asan_interceptors.h +++ b/contrib/compiler-rt/lib/asan/asan_interceptors.h @@ -58,6 +58,12 @@ # define ASAN_INTERCEPT_SIGLONGJMP 0 #endif +#if SANITIZER_LINUX && !SANITIZER_ANDROID +# define ASAN_INTERCEPT___LONGJMP_CHK 1 +#else +# define ASAN_INTERCEPT___LONGJMP_CHK 0 +#endif + // Android bug: https://code.google.com/p/android/issues/detail?id=61799 #if ASAN_HAS_EXCEPTIONS && !SANITIZER_WINDOWS && \ !(SANITIZER_ANDROID && defined(__i386)) diff --git a/contrib/compiler-rt/lib/asan/asan_interface.inc b/contrib/compiler-rt/lib/asan/asan_interface.inc new file mode 100644 index 0000000..e65f617 --- /dev/null +++ b/contrib/compiler-rt/lib/asan/asan_interface.inc @@ -0,0 +1,169 @@ +//===-- asan_interface.inc ------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// Asan interface list. +//===----------------------------------------------------------------------===// +INTERFACE_FUNCTION(__asan_addr_is_in_fake_stack) +INTERFACE_FUNCTION(__asan_address_is_poisoned) +INTERFACE_FUNCTION(__asan_after_dynamic_init) +INTERFACE_FUNCTION(__asan_alloca_poison) +INTERFACE_FUNCTION(__asan_allocas_unpoison) +INTERFACE_FUNCTION(__asan_before_dynamic_init) +INTERFACE_FUNCTION(__asan_describe_address) +INTERFACE_FUNCTION(__asan_exp_load1) +INTERFACE_FUNCTION(__asan_exp_load2) +INTERFACE_FUNCTION(__asan_exp_load4) +INTERFACE_FUNCTION(__asan_exp_load8) +INTERFACE_FUNCTION(__asan_exp_load16) +INTERFACE_FUNCTION(__asan_exp_loadN) +INTERFACE_FUNCTION(__asan_exp_store1) +INTERFACE_FUNCTION(__asan_exp_store2) +INTERFACE_FUNCTION(__asan_exp_store4) +INTERFACE_FUNCTION(__asan_exp_store8) +INTERFACE_FUNCTION(__asan_exp_store16) +INTERFACE_FUNCTION(__asan_exp_storeN) +INTERFACE_FUNCTION(__asan_get_alloc_stack) +INTERFACE_FUNCTION(__asan_get_current_fake_stack) +INTERFACE_FUNCTION(__asan_get_free_stack) +INTERFACE_FUNCTION(__asan_get_report_access_size) +INTERFACE_FUNCTION(__asan_get_report_access_type) +INTERFACE_FUNCTION(__asan_get_report_address) +INTERFACE_FUNCTION(__asan_get_report_bp) +INTERFACE_FUNCTION(__asan_get_report_description) +INTERFACE_FUNCTION(__asan_get_report_pc) +INTERFACE_FUNCTION(__asan_get_report_sp) +INTERFACE_FUNCTION(__asan_get_shadow_mapping) +INTERFACE_FUNCTION(__asan_handle_no_return) +INTERFACE_FUNCTION(__asan_init) +INTERFACE_FUNCTION(__asan_load_cxx_array_cookie) +INTERFACE_FUNCTION(__asan_load1) +INTERFACE_FUNCTION(__asan_load2) +INTERFACE_FUNCTION(__asan_load4) +INTERFACE_FUNCTION(__asan_load8) +INTERFACE_FUNCTION(__asan_load16) +INTERFACE_FUNCTION(__asan_loadN) +INTERFACE_FUNCTION(__asan_load1_noabort) +INTERFACE_FUNCTION(__asan_load2_noabort) +INTERFACE_FUNCTION(__asan_load4_noabort) +INTERFACE_FUNCTION(__asan_load8_noabort) +INTERFACE_FUNCTION(__asan_load16_noabort) +INTERFACE_FUNCTION(__asan_loadN_noabort) +INTERFACE_FUNCTION(__asan_locate_address) +INTERFACE_FUNCTION(__asan_memcpy) +INTERFACE_FUNCTION(__asan_memmove) +INTERFACE_FUNCTION(__asan_memset) +INTERFACE_FUNCTION(__asan_poison_cxx_array_cookie) +INTERFACE_FUNCTION(__asan_poison_intra_object_redzone) +INTERFACE_FUNCTION(__asan_poison_memory_region) +INTERFACE_FUNCTION(__asan_poison_stack_memory) +INTERFACE_FUNCTION(__asan_print_accumulated_stats) +INTERFACE_FUNCTION(__asan_region_is_poisoned) +INTERFACE_FUNCTION(__asan_register_globals) +INTERFACE_FUNCTION(__asan_register_elf_globals) +INTERFACE_FUNCTION(__asan_register_image_globals) +INTERFACE_FUNCTION(__asan_report_error) +INTERFACE_FUNCTION(__asan_report_exp_load1) +INTERFACE_FUNCTION(__asan_report_exp_load2) +INTERFACE_FUNCTION(__asan_report_exp_load4) +INTERFACE_FUNCTION(__asan_report_exp_load8) +INTERFACE_FUNCTION(__asan_report_exp_load16) +INTERFACE_FUNCTION(__asan_report_exp_load_n) +INTERFACE_FUNCTION(__asan_report_exp_store1) +INTERFACE_FUNCTION(__asan_report_exp_store2) +INTERFACE_FUNCTION(__asan_report_exp_store4) +INTERFACE_FUNCTION(__asan_report_exp_store8) +INTERFACE_FUNCTION(__asan_report_exp_store16) +INTERFACE_FUNCTION(__asan_report_exp_store_n) +INTERFACE_FUNCTION(__asan_report_load1) +INTERFACE_FUNCTION(__asan_report_load2) +INTERFACE_FUNCTION(__asan_report_load4) +INTERFACE_FUNCTION(__asan_report_load8) +INTERFACE_FUNCTION(__asan_report_load16) +INTERFACE_FUNCTION(__asan_report_load_n) +INTERFACE_FUNCTION(__asan_report_load1_noabort) +INTERFACE_FUNCTION(__asan_report_load2_noabort) +INTERFACE_FUNCTION(__asan_report_load4_noabort) +INTERFACE_FUNCTION(__asan_report_load8_noabort) +INTERFACE_FUNCTION(__asan_report_load16_noabort) +INTERFACE_FUNCTION(__asan_report_load_n_noabort) +INTERFACE_FUNCTION(__asan_report_present) +INTERFACE_FUNCTION(__asan_report_store1) +INTERFACE_FUNCTION(__asan_report_store2) +INTERFACE_FUNCTION(__asan_report_store4) +INTERFACE_FUNCTION(__asan_report_store8) +INTERFACE_FUNCTION(__asan_report_store16) +INTERFACE_FUNCTION(__asan_report_store_n) +INTERFACE_FUNCTION(__asan_report_store1_noabort) +INTERFACE_FUNCTION(__asan_report_store2_noabort) +INTERFACE_FUNCTION(__asan_report_store4_noabort) +INTERFACE_FUNCTION(__asan_report_store8_noabort) +INTERFACE_FUNCTION(__asan_report_store16_noabort) +INTERFACE_FUNCTION(__asan_report_store_n_noabort) +INTERFACE_FUNCTION(__asan_set_death_callback) +INTERFACE_FUNCTION(__asan_set_error_report_callback) +INTERFACE_FUNCTION(__asan_set_shadow_00) +INTERFACE_FUNCTION(__asan_set_shadow_f1) +INTERFACE_FUNCTION(__asan_set_shadow_f2) +INTERFACE_FUNCTION(__asan_set_shadow_f3) +INTERFACE_FUNCTION(__asan_set_shadow_f5) +INTERFACE_FUNCTION(__asan_set_shadow_f8) +INTERFACE_FUNCTION(__asan_stack_free_0) +INTERFACE_FUNCTION(__asan_stack_free_1) +INTERFACE_FUNCTION(__asan_stack_free_2) +INTERFACE_FUNCTION(__asan_stack_free_3) +INTERFACE_FUNCTION(__asan_stack_free_4) +INTERFACE_FUNCTION(__asan_stack_free_5) +INTERFACE_FUNCTION(__asan_stack_free_6) +INTERFACE_FUNCTION(__asan_stack_free_7) +INTERFACE_FUNCTION(__asan_stack_free_8) +INTERFACE_FUNCTION(__asan_stack_free_9) +INTERFACE_FUNCTION(__asan_stack_free_10) +INTERFACE_FUNCTION(__asan_stack_malloc_0) +INTERFACE_FUNCTION(__asan_stack_malloc_1) +INTERFACE_FUNCTION(__asan_stack_malloc_2) +INTERFACE_FUNCTION(__asan_stack_malloc_3) +INTERFACE_FUNCTION(__asan_stack_malloc_4) +INTERFACE_FUNCTION(__asan_stack_malloc_5) +INTERFACE_FUNCTION(__asan_stack_malloc_6) +INTERFACE_FUNCTION(__asan_stack_malloc_7) +INTERFACE_FUNCTION(__asan_stack_malloc_8) +INTERFACE_FUNCTION(__asan_stack_malloc_9) +INTERFACE_FUNCTION(__asan_stack_malloc_10) +INTERFACE_FUNCTION(__asan_store1) +INTERFACE_FUNCTION(__asan_store2) +INTERFACE_FUNCTION(__asan_store4) +INTERFACE_FUNCTION(__asan_store8) +INTERFACE_FUNCTION(__asan_store16) +INTERFACE_FUNCTION(__asan_storeN) +INTERFACE_FUNCTION(__asan_store1_noabort) +INTERFACE_FUNCTION(__asan_store2_noabort) +INTERFACE_FUNCTION(__asan_store4_noabort) +INTERFACE_FUNCTION(__asan_store8_noabort) +INTERFACE_FUNCTION(__asan_store16_noabort) +INTERFACE_FUNCTION(__asan_storeN_noabort) +INTERFACE_FUNCTION(__asan_unpoison_intra_object_redzone) +INTERFACE_FUNCTION(__asan_unpoison_memory_region) +INTERFACE_FUNCTION(__asan_unpoison_stack_memory) +INTERFACE_FUNCTION(__asan_unregister_globals) +INTERFACE_FUNCTION(__asan_unregister_elf_globals) +INTERFACE_FUNCTION(__asan_unregister_image_globals) +INTERFACE_FUNCTION(__asan_version_mismatch_check_v8) +INTERFACE_FUNCTION(__sanitizer_finish_switch_fiber) +INTERFACE_FUNCTION(__sanitizer_print_stack_trace) +INTERFACE_FUNCTION(__sanitizer_ptr_cmp) +INTERFACE_FUNCTION(__sanitizer_ptr_sub) +INTERFACE_FUNCTION(__sanitizer_start_switch_fiber) +INTERFACE_FUNCTION(__sanitizer_unaligned_load16) +INTERFACE_FUNCTION(__sanitizer_unaligned_load32) +INTERFACE_FUNCTION(__sanitizer_unaligned_load64) +INTERFACE_FUNCTION(__sanitizer_unaligned_store16) +INTERFACE_FUNCTION(__sanitizer_unaligned_store32) +INTERFACE_FUNCTION(__sanitizer_unaligned_store64) +INTERFACE_WEAK_FUNCTION(__asan_default_options) +INTERFACE_WEAK_FUNCTION(__asan_default_suppressions) +INTERFACE_WEAK_FUNCTION(__asan_on_error) diff --git a/contrib/compiler-rt/lib/asan/asan_interface_internal.h b/contrib/compiler-rt/lib/asan/asan_interface_internal.h index 8cd424c..b974c0c 100644 --- a/contrib/compiler-rt/lib/asan/asan_interface_internal.h +++ b/contrib/compiler-rt/lib/asan/asan_interface_internal.h @@ -67,6 +67,11 @@ extern "C" { SANITIZER_INTERFACE_ATTRIBUTE void __asan_unregister_image_globals(uptr *flag); + SANITIZER_INTERFACE_ATTRIBUTE + void __asan_register_elf_globals(uptr *flag, void *start, void *stop); + SANITIZER_INTERFACE_ATTRIBUTE + void __asan_unregister_elf_globals(uptr *flag, void *start, void *stop); + // These two functions should be called by the instrumented code. // 'globals' is an array of structures describing 'n' globals. SANITIZER_INTERFACE_ATTRIBUTE @@ -165,12 +170,12 @@ extern "C" { void __asan_set_error_report_callback(void (*callback)(const char*)); SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE - /* OPTIONAL */ void __asan_on_error(); + void __asan_on_error(); SANITIZER_INTERFACE_ATTRIBUTE void __asan_print_accumulated_stats(); SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE - /* OPTIONAL */ const char* __asan_default_options(); + const char* __asan_default_options(); SANITIZER_INTERFACE_ATTRIBUTE extern uptr __asan_shadow_memory_dynamic_address; @@ -242,6 +247,9 @@ extern "C" { void __asan_alloca_poison(uptr addr, uptr size); SANITIZER_INTERFACE_ATTRIBUTE void __asan_allocas_unpoison(uptr top, uptr bottom); + + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + const char* __asan_default_suppressions(); } // extern "C" #endif // ASAN_INTERFACE_INTERNAL_H diff --git a/contrib/compiler-rt/lib/asan/asan_internal.h b/contrib/compiler-rt/lib/asan/asan_internal.h index 1dc678c..f09bbd8 100644 --- a/contrib/compiler-rt/lib/asan/asan_internal.h +++ b/contrib/compiler-rt/lib/asan/asan_internal.h @@ -64,9 +64,9 @@ void AsanInitFromRtl(); // asan_win.cc void InitializePlatformExceptionHandlers(); - -// asan_win.cc / asan_posix.cc -const char *DescribeSignalOrException(int signo); +// Returns whether an address is a valid allocated system heap block. +// 'addr' must point to the beginning of the block. +bool IsSystemHeapAddress(uptr addr); // asan_rtl.cc void NORETURN ShowStatsAndAbort(); @@ -75,6 +75,7 @@ void NORETURN ShowStatsAndAbort(); void ReplaceSystemMalloc(); // asan_linux.cc / asan_mac.cc / asan_win.cc +uptr FindDynamicShadowStart(); void *AsanDoesNotSupportStaticLinkage(); void AsanCheckDynamicRTPrereqs(); void AsanCheckIncompatibleRT(); diff --git a/contrib/compiler-rt/lib/asan/asan_linux.cc b/contrib/compiler-rt/lib/asan/asan_linux.cc index c051573..6d47ba4 100644 --- a/contrib/compiler-rt/lib/asan/asan_linux.cc +++ b/contrib/compiler-rt/lib/asan/asan_linux.cc @@ -70,12 +70,18 @@ namespace __asan { void InitializePlatformInterceptors() {} void InitializePlatformExceptionHandlers() {} +bool IsSystemHeapAddress (uptr addr) { return false; } void *AsanDoesNotSupportStaticLinkage() { // This will fail to link with -static. return &_DYNAMIC; // defined in link.h } +uptr FindDynamicShadowStart() { + UNREACHABLE("FindDynamicShadowStart is not available"); + return 0; +} + void AsanApplyToGlobals(globals_op_fptr op, const void *needle) { UNIMPLEMENTED(); } @@ -110,7 +116,7 @@ static void ReportIncompatibleRT() { } void AsanCheckDynamicRTPrereqs() { - if (!ASAN_DYNAMIC) + if (!ASAN_DYNAMIC || !flags()->verify_asan_link_order) return; // Ensure that dynamic RT is the first DSO in the list @@ -139,9 +145,9 @@ void AsanCheckIncompatibleRT() { // system libraries, causing crashes later in ASan initialization. MemoryMappingLayout proc_maps(/*cache_enabled*/true); char filename[128]; - while (proc_maps.Next(nullptr, nullptr, nullptr, filename, - sizeof(filename), nullptr)) { - if (IsDynamicRTName(filename)) { + MemoryMappedSegment segment(filename, sizeof(filename)); + while (proc_maps.Next(&segment)) { + if (IsDynamicRTName(segment.filename)) { Report("Your application is linked against " "incompatible ASan runtimes.\n"); Die(); diff --git a/contrib/compiler-rt/lib/asan/asan_mac.cc b/contrib/compiler-rt/lib/asan/asan_mac.cc index baf533a..b7af1a5 100644 --- a/contrib/compiler-rt/lib/asan/asan_mac.cc +++ b/contrib/compiler-rt/lib/asan/asan_mac.cc @@ -48,12 +48,36 @@ namespace __asan { void InitializePlatformInterceptors() {} void InitializePlatformExceptionHandlers() {} +bool IsSystemHeapAddress (uptr addr) { return false; } // No-op. Mac does not support static linkage anyway. void *AsanDoesNotSupportStaticLinkage() { return 0; } +uptr FindDynamicShadowStart() { + uptr granularity = GetMmapGranularity(); + uptr alignment = 8 * granularity; + uptr left_padding = granularity; + uptr space_size = kHighShadowEnd + left_padding; + + uptr largest_gap_found = 0; + uptr shadow_start = FindAvailableMemoryRange(space_size, alignment, + granularity, &largest_gap_found); + // If the shadow doesn't fit, restrict the address space to make it fit. + if (shadow_start == 0) { + uptr new_max_vm = RoundDownTo(largest_gap_found << SHADOW_SCALE, alignment); + RestrictMemoryToMaxAddress(new_max_vm); + kHighMemEnd = new_max_vm - 1; + space_size = kHighShadowEnd + left_padding; + shadow_start = + FindAvailableMemoryRange(space_size, alignment, granularity, nullptr); + } + CHECK_NE((uptr)0, shadow_start); + CHECK(IsAligned(shadow_start, alignment)); + return shadow_start; +} + // No-op. Mac does not support static linkage anyway. void AsanCheckDynamicRTPrereqs() {} @@ -138,7 +162,8 @@ void asan_register_worker_thread(int parent_tid, StackTrace *stack) { t = AsanThread::Create(/* start_routine */ nullptr, /* arg */ nullptr, parent_tid, stack, /* detached */ true); t->Init(); - asanThreadRegistry().StartThread(t->tid(), 0, 0); + asanThreadRegistry().StartThread(t->tid(), GetTid(), + /* workerthread */ true, 0); SetCurrentThread(t); } } diff --git a/contrib/compiler-rt/lib/asan/asan_malloc_linux.cc b/contrib/compiler-rt/lib/asan/asan_malloc_linux.cc index 8c99d3b..fd40f47 100644 --- a/contrib/compiler-rt/lib/asan/asan_malloc_linux.cc +++ b/contrib/compiler-rt/lib/asan/asan_malloc_linux.cc @@ -60,36 +60,42 @@ INTERCEPTOR(void, cfree, void *ptr) { #endif // SANITIZER_INTERCEPT_CFREE INTERCEPTOR(void*, malloc, uptr size) { - if (UNLIKELY(!asan_inited)) + if (UNLIKELY(asan_init_is_running)) // Hack: dlsym calls malloc before REAL(malloc) is retrieved from dlsym. return AllocateFromLocalPool(size); + ENSURE_ASAN_INITED(); GET_STACK_TRACE_MALLOC; return asan_malloc(size, &stack); } INTERCEPTOR(void*, calloc, uptr nmemb, uptr size) { - if (UNLIKELY(!asan_inited)) + if (UNLIKELY(asan_init_is_running)) // Hack: dlsym calls calloc before REAL(calloc) is retrieved from dlsym. return AllocateFromLocalPool(nmemb * size); + ENSURE_ASAN_INITED(); GET_STACK_TRACE_MALLOC; return asan_calloc(nmemb, size, &stack); } INTERCEPTOR(void*, realloc, void *ptr, uptr size) { - GET_STACK_TRACE_MALLOC; if (UNLIKELY(IsInDlsymAllocPool(ptr))) { - uptr offset = (uptr)ptr - (uptr)alloc_memory_for_dlsym; - uptr copy_size = Min(size, kDlsymAllocPoolSize - offset); + const uptr offset = (uptr)ptr - (uptr)alloc_memory_for_dlsym; + const uptr copy_size = Min(size, kDlsymAllocPoolSize - offset); void *new_ptr; - if (UNLIKELY(!asan_inited)) { + if (UNLIKELY(asan_init_is_running)) { new_ptr = AllocateFromLocalPool(size); } else { - copy_size = size; - new_ptr = asan_malloc(copy_size, &stack); + ENSURE_ASAN_INITED(); + GET_STACK_TRACE_MALLOC; + new_ptr = asan_malloc(size, &stack); } internal_memcpy(new_ptr, ptr, copy_size); return new_ptr; } + if (UNLIKELY(asan_init_is_running)) + return AllocateFromLocalPool(size); + ENSURE_ASAN_INITED(); + GET_STACK_TRACE_MALLOC; return asan_realloc(ptr, size, &stack); } diff --git a/contrib/compiler-rt/lib/asan/asan_malloc_win.cc b/contrib/compiler-rt/lib/asan/asan_malloc_win.cc index 5163c04..efa0582 100644 --- a/contrib/compiler-rt/lib/asan/asan_malloc_win.cc +++ b/contrib/compiler-rt/lib/asan/asan_malloc_win.cc @@ -100,7 +100,7 @@ void *realloc(void *ptr, size_t size) { ALLOCATION_FUNCTION_ATTRIBUTE void *_realloc_dbg(void *ptr, size_t size, int) { - CHECK(!"_realloc_dbg should not exist!"); + UNREACHABLE("_realloc_dbg should not exist!"); return 0; } diff --git a/contrib/compiler-rt/lib/asan/asan_mapping.h b/contrib/compiler-rt/lib/asan/asan_mapping.h index d8e60a4..695740c 100644 --- a/contrib/compiler-rt/lib/asan/asan_mapping.h +++ b/contrib/compiler-rt/lib/asan/asan_mapping.h @@ -191,7 +191,6 @@ static const u64 kWindowsShadowOffset32 = 3ULL << 28; // 0x30000000 #define SHADOW_GRANULARITY (1ULL << SHADOW_SCALE) #define MEM_TO_SHADOW(mem) (((mem) >> SHADOW_SCALE) + (SHADOW_OFFSET)) -#define SHADOW_TO_MEM(shadow) (((shadow) - SHADOW_OFFSET) << SHADOW_SCALE) #define kLowMemBeg 0 #define kLowMemEnd (SHADOW_OFFSET ? SHADOW_OFFSET - 1 : 0) diff --git a/contrib/compiler-rt/lib/asan/asan_memory_profile.cc b/contrib/compiler-rt/lib/asan/asan_memory_profile.cc index c2678b9..05846c3 100644 --- a/contrib/compiler-rt/lib/asan/asan_memory_profile.cc +++ b/contrib/compiler-rt/lib/asan/asan_memory_profile.cc @@ -48,7 +48,7 @@ class HeapProfile { } } - void Print(uptr top_percent) { + void Print(uptr top_percent, uptr max_number_of_contexts) { InternalSort(&allocations_, allocations_.size(), [](const AllocationSite &a, const AllocationSite &b) { return a.total_size > b.total_size; @@ -57,12 +57,14 @@ class HeapProfile { uptr total_shown = 0; Printf("Live Heap Allocations: %zd bytes in %zd chunks; quarantined: " "%zd bytes in %zd chunks; %zd other chunks; total chunks: %zd; " - "showing top %zd%%\n", + "showing top %zd%% (at most %zd unique contexts)\n", total_allocated_user_size_, total_allocated_count_, total_quarantined_user_size_, total_quarantined_count_, total_other_count_, total_allocated_count_ + - total_quarantined_count_ + total_other_count_, top_percent); - for (uptr i = 0; i < allocations_.size(); i++) { + total_quarantined_count_ + total_other_count_, top_percent, + max_number_of_contexts); + for (uptr i = 0; i < Min(allocations_.size(), max_number_of_contexts); + i++) { auto &a = allocations_[i]; Printf("%zd byte(s) (%zd%%) in %zd allocation(s)\n", a.total_size, a.total_size * 100 / total_allocated_user_size_, a.count); @@ -103,16 +105,23 @@ static void MemoryProfileCB(const SuspendedThreadsList &suspended_threads_list, void *argument) { HeapProfile hp; __lsan::ForEachChunk(ChunkCallback, &hp); - hp.Print(reinterpret_cast<uptr>(argument)); + uptr *Arg = reinterpret_cast<uptr*>(argument); + hp.Print(Arg[0], Arg[1]); } } // namespace __asan +#endif // CAN_SANITIZE_LEAKS + extern "C" { SANITIZER_INTERFACE_ATTRIBUTE -void __sanitizer_print_memory_profile(uptr top_percent) { - __sanitizer::StopTheWorld(__asan::MemoryProfileCB, (void*)top_percent); +void __sanitizer_print_memory_profile(uptr top_percent, + uptr max_number_of_contexts) { +#if CAN_SANITIZE_LEAKS + uptr Arg[2]; + Arg[0] = top_percent; + Arg[1] = max_number_of_contexts; + __sanitizer::StopTheWorld(__asan::MemoryProfileCB, Arg); +#endif // CAN_SANITIZE_LEAKS } } // extern "C" - -#endif // CAN_SANITIZE_LEAKS diff --git a/contrib/compiler-rt/lib/asan/asan_new_delete.cc b/contrib/compiler-rt/lib/asan/asan_new_delete.cc index 3283fb3..e68c7f3 100644 --- a/contrib/compiler-rt/lib/asan/asan_new_delete.cc +++ b/contrib/compiler-rt/lib/asan/asan_new_delete.cc @@ -25,22 +25,26 @@ // dllexport would normally do. We need to export them in order to make the // VS2015 dynamic CRT (MD) work. #if SANITIZER_WINDOWS -# define CXX_OPERATOR_ATTRIBUTE -# ifdef _WIN64 -# pragma comment(linker, "/export:??2@YAPEAX_K@Z") // operator new -# pragma comment(linker, "/export:??3@YAXPEAX@Z") // operator delete -# pragma comment(linker, "/export:??3@YAXPEAX_K@Z") // sized operator delete -# pragma comment(linker, "/export:??_U@YAPEAX_K@Z") // operator new[] -# pragma comment(linker, "/export:??_V@YAXPEAX@Z") // operator delete[] -# else -# pragma comment(linker, "/export:??2@YAPAXI@Z") // operator new -# pragma comment(linker, "/export:??3@YAXPAX@Z") // operator delete -# pragma comment(linker, "/export:??3@YAXPAXI@Z") // sized operator delete -# pragma comment(linker, "/export:??_U@YAPAXI@Z") // operator new[] -# pragma comment(linker, "/export:??_V@YAXPAX@Z") // operator delete[] -# endif +#define CXX_OPERATOR_ATTRIBUTE +#define COMMENT_EXPORT(sym) __pragma(comment(linker, "/export:" sym)) +#ifdef _WIN64 +COMMENT_EXPORT("??2@YAPEAX_K@Z") // operator new +COMMENT_EXPORT("??2@YAPEAX_KAEBUnothrow_t@std@@@Z") // operator new nothrow +COMMENT_EXPORT("??3@YAXPEAX@Z") // operator delete +COMMENT_EXPORT("??3@YAXPEAX_K@Z") // sized operator delete +COMMENT_EXPORT("??_U@YAPEAX_K@Z") // operator new[] +COMMENT_EXPORT("??_V@YAXPEAX@Z") // operator delete[] #else -# define CXX_OPERATOR_ATTRIBUTE INTERCEPTOR_ATTRIBUTE +COMMENT_EXPORT("??2@YAPAXI@Z") // operator new +COMMENT_EXPORT("??2@YAPAXIABUnothrow_t@std@@@Z") // operator new nothrow +COMMENT_EXPORT("??3@YAXPAX@Z") // operator delete +COMMENT_EXPORT("??3@YAXPAXI@Z") // sized operator delete +COMMENT_EXPORT("??_U@YAPAXI@Z") // operator new[] +COMMENT_EXPORT("??_V@YAXPAX@Z") // operator delete[] +#endif +#undef COMMENT_EXPORT +#else +#define CXX_OPERATOR_ATTRIBUTE INTERCEPTOR_ATTRIBUTE #endif using namespace __asan; // NOLINT @@ -63,12 +67,17 @@ struct nothrow_t {}; enum class align_val_t: size_t {}; } // namespace std -#define OPERATOR_NEW_BODY(type) \ +// TODO(alekseys): throw std::bad_alloc instead of dying on OOM. +#define OPERATOR_NEW_BODY(type, nothrow) \ GET_STACK_TRACE_MALLOC;\ - return asan_memalign(0, size, &stack, type); -#define OPERATOR_NEW_BODY_ALIGN(type) \ + void *res = asan_memalign(0, size, &stack, type);\ + if (!nothrow && UNLIKELY(!res)) DieOnFailure::OnOOM();\ + return res; +#define OPERATOR_NEW_BODY_ALIGN(type, nothrow) \ GET_STACK_TRACE_MALLOC;\ - return asan_memalign((uptr)align, size, &stack, type); + void *res = asan_memalign((uptr)align, size, &stack, type);\ + if (!nothrow && UNLIKELY(!res)) DieOnFailure::OnOOM();\ + return res; // On OS X it's not enough to just provide our own 'operator new' and // 'operator delete' implementations, because they're going to be in the @@ -79,40 +88,42 @@ enum class align_val_t: size_t {}; // OS X we need to intercept them using their mangled names. #if !SANITIZER_MAC CXX_OPERATOR_ATTRIBUTE -void *operator new(size_t size) { OPERATOR_NEW_BODY(FROM_NEW); } +void *operator new(size_t size) +{ OPERATOR_NEW_BODY(FROM_NEW, false /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE -void *operator new[](size_t size) { OPERATOR_NEW_BODY(FROM_NEW_BR); } +void *operator new[](size_t size) +{ OPERATOR_NEW_BODY(FROM_NEW_BR, false /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new(size_t size, std::nothrow_t const&) -{ OPERATOR_NEW_BODY(FROM_NEW); } +{ OPERATOR_NEW_BODY(FROM_NEW, true /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new[](size_t size, std::nothrow_t const&) -{ OPERATOR_NEW_BODY(FROM_NEW_BR); } +{ OPERATOR_NEW_BODY(FROM_NEW_BR, true /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new(size_t size, std::align_val_t align) -{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW); } +{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW, false /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new[](size_t size, std::align_val_t align) -{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW_BR); } +{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW_BR, false /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new(size_t size, std::align_val_t align, std::nothrow_t const&) -{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW); } +{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW, true /*nothrow*/); } CXX_OPERATOR_ATTRIBUTE void *operator new[](size_t size, std::align_val_t align, std::nothrow_t const&) -{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW_BR); } +{ OPERATOR_NEW_BODY_ALIGN(FROM_NEW_BR, true /*nothrow*/); } #else // SANITIZER_MAC INTERCEPTOR(void *, _Znwm, size_t size) { - OPERATOR_NEW_BODY(FROM_NEW); + OPERATOR_NEW_BODY(FROM_NEW, false /*nothrow*/); } INTERCEPTOR(void *, _Znam, size_t size) { - OPERATOR_NEW_BODY(FROM_NEW_BR); + OPERATOR_NEW_BODY(FROM_NEW_BR, false /*nothrow*/); } INTERCEPTOR(void *, _ZnwmRKSt9nothrow_t, size_t size, std::nothrow_t const&) { - OPERATOR_NEW_BODY(FROM_NEW); + OPERATOR_NEW_BODY(FROM_NEW, true /*nothrow*/); } INTERCEPTOR(void *, _ZnamRKSt9nothrow_t, size_t size, std::nothrow_t const&) { - OPERATOR_NEW_BODY(FROM_NEW_BR); + OPERATOR_NEW_BODY(FROM_NEW_BR, true /*nothrow*/); } #endif diff --git a/contrib/compiler-rt/lib/asan/asan_posix.cc b/contrib/compiler-rt/lib/asan/asan_posix.cc index 8e56763..added74 100644 --- a/contrib/compiler-rt/lib/asan/asan_posix.cc +++ b/contrib/compiler-rt/lib/asan/asan_posix.cc @@ -33,19 +33,6 @@ namespace __asan { -const char *DescribeSignalOrException(int signo) { - switch (signo) { - case SIGFPE: - return "FPE"; - case SIGILL: - return "ILL"; - case SIGABRT: - return "ABRT"; - default: - return "SEGV"; - } -} - void AsanOnDeadlySignal(int signo, void *siginfo, void *context) { ScopedDeadlySignal signal_scope(GetCurrentThread()); int code = (int)((siginfo_t*)siginfo)->si_code; @@ -72,7 +59,7 @@ void AsanOnDeadlySignal(int signo, void *siginfo, void *context) { // lis r0,-10000 // stdux r1,r1,r0 # store sp to [sp-10000] and update sp by -10000 // If the store faults then sp will not have been updated, so test above - // will not work, becase the fault address will be more than just "slightly" + // will not work, because the fault address will be more than just "slightly" // below sp. if (!IsStackAccess && IsAccessibleMemoryRange(sig.pc, 4)) { u32 inst = *(unsigned *)sig.pc; diff --git a/contrib/compiler-rt/lib/asan/asan_report.cc b/contrib/compiler-rt/lib/asan/asan_report.cc index 3ad48fa..2e477f2 100644 --- a/contrib/compiler-rt/lib/asan/asan_report.cc +++ b/contrib/compiler-rt/lib/asan/asan_report.cc @@ -88,7 +88,8 @@ bool ParseFrameDescription(const char *frame_descr, char *p; // This string is created by the compiler and has the following form: // "n alloc_1 alloc_2 ... alloc_n" - // where alloc_i looks like "offset size len ObjectName". + // where alloc_i looks like "offset size len ObjectName" + // or "offset size len ObjectName:line". uptr n_objects = (uptr)internal_simple_strtoll(frame_descr, &p, 10); if (n_objects == 0) return false; @@ -101,7 +102,14 @@ bool ParseFrameDescription(const char *frame_descr, return false; } p++; - StackVarDescr var = {beg, size, p, len}; + char *colon_pos = internal_strchr(p, ':'); + uptr line = 0; + uptr name_len = len; + if (colon_pos != nullptr && colon_pos < p + len) { + name_len = colon_pos - p; + line = (uptr)internal_simple_strtoll(colon_pos + 1, nullptr, 10); + } + StackVarDescr var = {beg, size, p, name_len, line}; vars->push_back(var); p += len; } @@ -196,6 +204,14 @@ class ScopedInErrorReport { error_report_callback(buffer_copy.data()); } + if (halt_on_error_ && common_flags()->abort_on_error) { + // On Android the message is truncated to 512 characters. + // FIXME: implement "compact" error format, possibly without, or with + // highly compressed stack traces? + // FIXME: or just use the summary line as abort message? + SetAbortMessage(buffer_copy.data()); + } + // In halt_on_error = false mode, reset the current error object (before // unlocking). if (!halt_on_error_) @@ -488,9 +504,6 @@ void __sanitizer_ptr_cmp(void *a, void *b) { } } // extern "C" -#if !SANITIZER_SUPPORTS_WEAK_HOOKS // Provide default implementation of __asan_on_error that does nothing // and may be overriden by user. -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE NOINLINE -void __asan_on_error() {} -#endif +SANITIZER_INTERFACE_WEAK_DEF(void, __asan_on_error, void) {} diff --git a/contrib/compiler-rt/lib/asan/asan_report.h b/contrib/compiler-rt/lib/asan/asan_report.h index 5ebfda6..5a3533a 100644 --- a/contrib/compiler-rt/lib/asan/asan_report.h +++ b/contrib/compiler-rt/lib/asan/asan_report.h @@ -23,6 +23,7 @@ struct StackVarDescr { uptr size; const char *name_pos; uptr name_len; + uptr line; }; // Returns the number of globals close to the provided address and copies diff --git a/contrib/compiler-rt/lib/asan/asan_rtl.cc b/contrib/compiler-rt/lib/asan/asan_rtl.cc index d9d7d7e..5ae3568 100644 --- a/contrib/compiler-rt/lib/asan/asan_rtl.cc +++ b/contrib/compiler-rt/lib/asan/asan_rtl.cc @@ -438,15 +438,7 @@ static void InitializeShadowMemory() { if (shadow_start == kDefaultShadowSentinel) { __asan_shadow_memory_dynamic_address = 0; CHECK_EQ(0, kLowShadowBeg); - - uptr granularity = GetMmapGranularity(); - uptr alignment = 8 * granularity; - uptr left_padding = granularity; - uptr space_size = kHighShadowEnd + left_padding; - - shadow_start = FindAvailableMemoryRange(space_size, alignment, granularity); - CHECK_NE((uptr)0, shadow_start); - CHECK(IsAligned(shadow_start, alignment)); + shadow_start = FindDynamicShadowStart(); } // Update the shadow memory address (potentially) used by instrumentation. __asan_shadow_memory_dynamic_address = shadow_start; diff --git a/contrib/compiler-rt/lib/asan/asan_suppressions.cc b/contrib/compiler-rt/lib/asan/asan_suppressions.cc index 62c868d..ac8aa02 100644 --- a/contrib/compiler-rt/lib/asan/asan_suppressions.cc +++ b/contrib/compiler-rt/lib/asan/asan_suppressions.cc @@ -31,15 +31,9 @@ static const char *kSuppressionTypes[] = { kInterceptorName, kInterceptorViaFunction, kInterceptorViaLibrary, kODRViolation}; -extern "C" { -#if SANITIZER_SUPPORTS_WEAK_HOOKS -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -const char *__asan_default_suppressions(); -#else -// No week hooks, provide empty implementation. -const char *__asan_default_suppressions() { return ""; } -#endif // SANITIZER_SUPPORTS_WEAK_HOOKS -} // extern "C" +SANITIZER_INTERFACE_WEAK_DEF(const char *, __asan_default_suppressions, void) { + return ""; +} void InitializeSuppressions() { CHECK_EQ(nullptr, suppression_ctx); diff --git a/contrib/compiler-rt/lib/asan/asan_thread.cc b/contrib/compiler-rt/lib/asan/asan_thread.cc index 537b53d..b1a0d9a 100644 --- a/contrib/compiler-rt/lib/asan/asan_thread.cc +++ b/contrib/compiler-rt/lib/asan/asan_thread.cc @@ -166,16 +166,19 @@ void AsanThread::FinishSwitchFiber(FakeStack *fake_stack_save, } inline AsanThread::StackBounds AsanThread::GetStackBounds() const { - if (!atomic_load(&stack_switching_, memory_order_acquire)) - return StackBounds{stack_bottom_, stack_top_}; // NOLINT + if (!atomic_load(&stack_switching_, memory_order_acquire)) { + // Make sure the stack bounds are fully initialized. + if (stack_bottom_ >= stack_top_) return {0, 0}; + return {stack_bottom_, stack_top_}; + } char local; const uptr cur_stack = (uptr)&local; // Note: need to check next stack first, because FinishSwitchFiber // may be in process of overwriting stack_top_/bottom_. But in such case // we are already on the next stack. if (cur_stack >= next_stack_bottom_ && cur_stack < next_stack_top_) - return StackBounds{next_stack_bottom_, next_stack_top_}; // NOLINT - return StackBounds{stack_bottom_, stack_top_}; // NOLINT + return {next_stack_bottom_, next_stack_top_}; + return {stack_bottom_, stack_top_}; } uptr AsanThread::stack_top() { @@ -237,9 +240,10 @@ void AsanThread::Init() { } thread_return_t AsanThread::ThreadStart( - uptr os_id, atomic_uintptr_t *signal_thread_is_registered) { + tid_t os_id, atomic_uintptr_t *signal_thread_is_registered) { Init(); - asanThreadRegistry().StartThread(tid(), os_id, nullptr); + asanThreadRegistry().StartThread(tid(), os_id, /*workerthread*/ false, + nullptr); if (signal_thread_is_registered) atomic_store(signal_thread_is_registered, 1, memory_order_release); @@ -299,24 +303,27 @@ bool AsanThread::GetStackFrameAccessByAddr(uptr addr, return true; } uptr aligned_addr = addr & ~(SANITIZER_WORDSIZE/8 - 1); // align addr. + uptr mem_ptr = RoundDownTo(aligned_addr, SHADOW_GRANULARITY); u8 *shadow_ptr = (u8*)MemToShadow(aligned_addr); u8 *shadow_bottom = (u8*)MemToShadow(bottom); while (shadow_ptr >= shadow_bottom && *shadow_ptr != kAsanStackLeftRedzoneMagic) { shadow_ptr--; + mem_ptr -= SHADOW_GRANULARITY; } while (shadow_ptr >= shadow_bottom && *shadow_ptr == kAsanStackLeftRedzoneMagic) { shadow_ptr--; + mem_ptr -= SHADOW_GRANULARITY; } if (shadow_ptr < shadow_bottom) { return false; } - uptr* ptr = (uptr*)SHADOW_TO_MEM((uptr)(shadow_ptr + 1)); + uptr* ptr = (uptr*)(mem_ptr + SHADOW_GRANULARITY); CHECK(ptr[0] == kCurrentStackFrameMagic); access->offset = addr - (uptr)ptr; access->frame_pc = ptr[2]; @@ -391,7 +398,7 @@ void EnsureMainThreadIDIsCorrect() { context->os_id = GetTid(); } -__asan::AsanThread *GetAsanThreadByOsIDLocked(uptr os_id) { +__asan::AsanThread *GetAsanThreadByOsIDLocked(tid_t os_id) { __asan::AsanThreadContext *context = static_cast<__asan::AsanThreadContext *>( __asan::asanThreadRegistry().FindThreadContextByOsIDLocked(os_id)); if (!context) return nullptr; @@ -401,7 +408,7 @@ __asan::AsanThread *GetAsanThreadByOsIDLocked(uptr os_id) { // --- Implementation of LSan-specific functions --- {{{1 namespace __lsan { -bool GetThreadRangesLocked(uptr os_id, uptr *stack_begin, uptr *stack_end, +bool GetThreadRangesLocked(tid_t os_id, uptr *stack_begin, uptr *stack_end, uptr *tls_begin, uptr *tls_end, uptr *cache_begin, uptr *cache_end, DTLS **dtls) { __asan::AsanThread *t = __asan::GetAsanThreadByOsIDLocked(os_id); @@ -417,7 +424,7 @@ bool GetThreadRangesLocked(uptr os_id, uptr *stack_begin, uptr *stack_end, return true; } -void ForEachExtraStackRange(uptr os_id, RangeIteratorCallback callback, +void ForEachExtraStackRange(tid_t os_id, RangeIteratorCallback callback, void *arg) { __asan::AsanThread *t = __asan::GetAsanThreadByOsIDLocked(os_id); if (t && t->has_fake_stack()) diff --git a/contrib/compiler-rt/lib/asan/asan_thread.h b/contrib/compiler-rt/lib/asan/asan_thread.h index f53dfb7..424f9e6 100644 --- a/contrib/compiler-rt/lib/asan/asan_thread.h +++ b/contrib/compiler-rt/lib/asan/asan_thread.h @@ -63,7 +63,7 @@ class AsanThread { void Destroy(); void Init(); // Should be called from the thread itself. - thread_return_t ThreadStart(uptr os_id, + thread_return_t ThreadStart(tid_t os_id, atomic_uintptr_t *signal_thread_is_registered); uptr stack_top(); diff --git a/contrib/compiler-rt/lib/asan/asan_win.cc b/contrib/compiler-rt/lib/asan/asan_win.cc index 78268d8..8a839d9 100644 --- a/contrib/compiler-rt/lib/asan/asan_win.cc +++ b/contrib/compiler-rt/lib/asan/asan_win.cc @@ -19,7 +19,6 @@ #include <stdlib.h> -#include "asan_globals_win.h" #include "asan_interceptors.h" #include "asan_internal.h" #include "asan_report.h" @@ -28,6 +27,8 @@ #include "asan_mapping.h" #include "sanitizer_common/sanitizer_libc.h" #include "sanitizer_common/sanitizer_mutex.h" +#include "sanitizer_common/sanitizer_win.h" +#include "sanitizer_common/sanitizer_win_defs.h" using namespace __asan; // NOLINT @@ -43,35 +44,50 @@ uptr __asan_get_shadow_memory_dynamic_address() { __asan_init(); return __asan_shadow_memory_dynamic_address; } - -// -------------------- A workaround for the absence of weak symbols ----- {{{ -// We don't have a direct equivalent of weak symbols when using MSVC, but we can -// use the /alternatename directive to tell the linker to default a specific -// symbol to a specific value, which works nicely for allocator hooks and -// __asan_default_options(). -void __sanitizer_default_malloc_hook(void *ptr, uptr size) { } -void __sanitizer_default_free_hook(void *ptr) { } -const char* __asan_default_default_options() { return ""; } -const char* __asan_default_default_suppressions() { return ""; } -void __asan_default_on_error() {} -// 64-bit msvc will not prepend an underscore for symbols. -#ifdef _WIN64 -#pragma comment(linker, "/alternatename:__sanitizer_malloc_hook=__sanitizer_default_malloc_hook") // NOLINT -#pragma comment(linker, "/alternatename:__sanitizer_free_hook=__sanitizer_default_free_hook") // NOLINT -#pragma comment(linker, "/alternatename:__asan_default_options=__asan_default_default_options") // NOLINT -#pragma comment(linker, "/alternatename:__asan_default_suppressions=__asan_default_default_suppressions") // NOLINT -#pragma comment(linker, "/alternatename:__asan_on_error=__asan_default_on_error") // NOLINT -#else -#pragma comment(linker, "/alternatename:___sanitizer_malloc_hook=___sanitizer_default_malloc_hook") // NOLINT -#pragma comment(linker, "/alternatename:___sanitizer_free_hook=___sanitizer_default_free_hook") // NOLINT -#pragma comment(linker, "/alternatename:___asan_default_options=___asan_default_default_options") // NOLINT -#pragma comment(linker, "/alternatename:___asan_default_suppressions=___asan_default_default_suppressions") // NOLINT -#pragma comment(linker, "/alternatename:___asan_on_error=___asan_default_on_error") // NOLINT -#endif -// }}} } // extern "C" // ---------------------- Windows-specific interceptors ---------------- {{{ +static LPTOP_LEVEL_EXCEPTION_FILTER default_seh_handler; +static LPTOP_LEVEL_EXCEPTION_FILTER user_seh_handler; + +extern "C" SANITIZER_INTERFACE_ATTRIBUTE +long __asan_unhandled_exception_filter(EXCEPTION_POINTERS *info) { + EXCEPTION_RECORD *exception_record = info->ExceptionRecord; + CONTEXT *context = info->ContextRecord; + + // FIXME: Handle EXCEPTION_STACK_OVERFLOW here. + + SignalContext sig = SignalContext::Create(exception_record, context); + ReportDeadlySignal(exception_record->ExceptionCode, sig); + UNREACHABLE("returned from reporting deadly signal"); +} + +// Wrapper SEH Handler. If the exception should be handled by asan, we call +// __asan_unhandled_exception_filter, otherwise, we execute the user provided +// exception handler or the default. +static long WINAPI SEHHandler(EXCEPTION_POINTERS *info) { + DWORD exception_code = info->ExceptionRecord->ExceptionCode; + if (__sanitizer::IsHandledDeadlyException(exception_code)) + return __asan_unhandled_exception_filter(info); + if (user_seh_handler) + return user_seh_handler(info); + // Bubble out to the default exception filter. + if (default_seh_handler) + return default_seh_handler(info); + return EXCEPTION_CONTINUE_SEARCH; +} + +INTERCEPTOR_WINAPI(LPTOP_LEVEL_EXCEPTION_FILTER, SetUnhandledExceptionFilter, + LPTOP_LEVEL_EXCEPTION_FILTER ExceptionFilter) { + CHECK(REAL(SetUnhandledExceptionFilter)); + if (ExceptionFilter == &SEHHandler) + return REAL(SetUnhandledExceptionFilter)(ExceptionFilter); + // We record the user provided exception handler to be called for all the + // exceptions unhandled by asan. + Swap(ExceptionFilter, user_seh_handler); + return ExceptionFilter; +} + INTERCEPTOR_WINAPI(void, RtlRaiseException, EXCEPTION_RECORD *ExceptionRecord) { CHECK(REAL(RtlRaiseException)); // This is a noreturn function, unless it's one of the exceptions raised to @@ -144,6 +160,7 @@ namespace __asan { void InitializePlatformInterceptors() { ASAN_INTERCEPT_FUNC(CreateThread); + ASAN_INTERCEPT_FUNC(SetUnhandledExceptionFilter); #ifdef _WIN64 ASAN_INTERCEPT_FUNC(__C_specific_handler); @@ -200,6 +217,18 @@ void *AsanDoesNotSupportStaticLinkage() { return 0; } +uptr FindDynamicShadowStart() { + uptr granularity = GetMmapGranularity(); + uptr alignment = 8 * granularity; + uptr left_padding = granularity; + uptr space_size = kHighShadowEnd + left_padding; + uptr shadow_start = + FindAvailableMemoryRange(space_size, alignment, granularity, nullptr); + CHECK_NE((uptr)0, shadow_start); + CHECK(IsAligned(shadow_start, alignment)); + return shadow_start; +} + void AsanCheckDynamicRTPrereqs() {} void AsanCheckIncompatibleRT() {} @@ -260,60 +289,8 @@ void InitializePlatformExceptionHandlers() { #endif } -static LPTOP_LEVEL_EXCEPTION_FILTER default_seh_handler; - -// Check based on flags if we should report this exception. -static bool ShouldReportDeadlyException(unsigned code) { - switch (code) { - case EXCEPTION_ACCESS_VIOLATION: - case EXCEPTION_IN_PAGE_ERROR: - return common_flags()->handle_segv; - case EXCEPTION_BREAKPOINT: - case EXCEPTION_ILLEGAL_INSTRUCTION: { - return common_flags()->handle_sigill; - } - } - return false; -} - -// Return the textual name for this exception. -const char *DescribeSignalOrException(int signo) { - unsigned code = signo; - // Get the string description of the exception if this is a known deadly - // exception. - switch (code) { - case EXCEPTION_ACCESS_VIOLATION: - return "access-violation"; - case EXCEPTION_IN_PAGE_ERROR: - return "in-page-error"; - case EXCEPTION_BREAKPOINT: - return "breakpoint"; - case EXCEPTION_ILLEGAL_INSTRUCTION: - return "illegal-instruction"; - } - return nullptr; -} - -extern "C" SANITIZER_INTERFACE_ATTRIBUTE -long __asan_unhandled_exception_filter(EXCEPTION_POINTERS *info) { - EXCEPTION_RECORD *exception_record = info->ExceptionRecord; - CONTEXT *context = info->ContextRecord; - - // Continue the search if the signal wasn't deadly. - if (!ShouldReportDeadlyException(exception_record->ExceptionCode)) - return EXCEPTION_CONTINUE_SEARCH; - // FIXME: Handle EXCEPTION_STACK_OVERFLOW here. - - SignalContext sig = SignalContext::Create(exception_record, context); - ReportDeadlySignal(exception_record->ExceptionCode, sig); - UNREACHABLE("returned from reporting deadly signal"); -} - -static long WINAPI SEHHandler(EXCEPTION_POINTERS *info) { - __asan_unhandled_exception_filter(info); - - // Bubble out to the default exception filter. - return default_seh_handler(info); +bool IsSystemHeapAddress(uptr addr) { + return ::HeapValidate(GetProcessHeap(), 0, (void*)addr) != FALSE; } // We want to install our own exception handler (EH) to print helpful reports @@ -368,7 +345,7 @@ __declspec(allocate(".CRT$XLAB")) void (NTAPI *__asan_tls_init)(void *, unsigned long, void *) = asan_thread_init; #endif -ASAN_LINK_GLOBALS_WIN() +WIN_FORCE_LINK(__asan_dso_reg_hook) // }}} } // namespace __asan diff --git a/contrib/compiler-rt/lib/asan/asan_win_dll_thunk.cc b/contrib/compiler-rt/lib/asan/asan_win_dll_thunk.cc index 4764fd0..c67116c 100644 --- a/contrib/compiler-rt/lib/asan/asan_win_dll_thunk.cc +++ b/contrib/compiler-rt/lib/asan/asan_win_dll_thunk.cc @@ -15,388 +15,41 @@ // See https://github.com/google/sanitizers/issues/209 for the details. //===----------------------------------------------------------------------===// -// Only compile this code when building asan_dll_thunk.lib -// Using #ifdef rather than relying on Makefiles etc. -// simplifies the build procedure. -#ifdef ASAN_DLL_THUNK +#ifdef SANITIZER_DLL_THUNK #include "asan_init_version.h" -#include "asan_globals_win.h" #include "interception/interception.h" +#include "sanitizer_common/sanitizer_win_defs.h" +#include "sanitizer_common/sanitizer_win_dll_thunk.h" #include "sanitizer_common/sanitizer_platform_interceptors.h" -#ifdef _M_IX86 -#define WINAPI __stdcall -#else -#define WINAPI -#endif - -// ---------- Function interception helper functions and macros ----------- {{{1 -extern "C" { -void *WINAPI GetModuleHandleA(const char *module_name); -void *WINAPI GetProcAddress(void *module, const char *proc_name); -void abort(); -} - -using namespace __sanitizer; - -static uptr getRealProcAddressOrDie(const char *name) { - uptr ret = - __interception::InternalGetProcAddress((void *)GetModuleHandleA(0), name); - if (!ret) - abort(); - return ret; -} - -// We need to intercept some functions (e.g. ASan interface, memory allocator -- -// let's call them "hooks") exported by the DLL thunk and forward the hooks to -// the runtime in the main module. -// However, we don't want to keep two lists of these hooks. -// To avoid that, the list of hooks should be defined using the -// INTERCEPT_WHEN_POSSIBLE macro. Then, all these hooks can be intercepted -// at once by calling INTERCEPT_HOOKS(). - -// Use macro+template magic to automatically generate the list of hooks. -// Each hook at line LINE defines a template class with a static -// FunctionInterceptor<LINE>::Execute() method intercepting the hook. -// The default implementation of FunctionInterceptor<LINE> is to call -// the Execute() method corresponding to the previous line. -template<int LINE> -struct FunctionInterceptor { - static void Execute() { FunctionInterceptor<LINE-1>::Execute(); } -}; - -// There shouldn't be any hooks with negative definition line number. -template<> -struct FunctionInterceptor<0> { - static void Execute() {} -}; - -#define INTERCEPT_WHEN_POSSIBLE(main_function, dll_function) \ - template <> struct FunctionInterceptor<__LINE__> { \ - static void Execute() { \ - uptr wrapper = getRealProcAddressOrDie(main_function); \ - if (!__interception::OverrideFunction((uptr)dll_function, wrapper, 0)) \ - abort(); \ - FunctionInterceptor<__LINE__ - 1>::Execute(); \ - } \ - }; - -// Special case of hooks -- ASan own interface functions. Those are only called -// after __asan_init, thus an empty implementation is sufficient. -#define INTERFACE_FUNCTION(name) \ - extern "C" __declspec(noinline) void name() { \ - volatile int prevent_icf = (__LINE__ << 8); (void)prevent_icf; \ - __debugbreak(); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name) - -// INTERCEPT_HOOKS must be used after the last INTERCEPT_WHEN_POSSIBLE. -#define INTERCEPT_HOOKS FunctionInterceptor<__LINE__>::Execute - -// We can't define our own version of strlen etc. because that would lead to -// link-time or even type mismatch errors. Instead, we can declare a function -// just to be able to get its address. Me may miss the first few calls to the -// functions since it can be called before __asan_init, but that would lead to -// false negatives in the startup code before user's global initializers, which -// isn't a big deal. -#define INTERCEPT_LIBRARY_FUNCTION(name) \ - extern "C" void name(); \ - INTERCEPT_WHEN_POSSIBLE(WRAPPER_NAME(name), name) - -// Disable compiler warnings that show up if we declare our own version -// of a compiler intrinsic (e.g. strlen). -#pragma warning(disable: 4391) -#pragma warning(disable: 4392) - -static void InterceptHooks(); -// }}} - -// ---------- Function wrapping helpers ----------------------------------- {{{1 -#define WRAP_V_V(name) \ - extern "C" void name() { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - fn(); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_V_W(name) \ - extern "C" void name(void *arg) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - fn(arg); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_V_WW(name) \ - extern "C" void name(void *arg1, void *arg2) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - fn(arg1, arg2); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_V_WWW(name) \ - extern "C" void name(void *arg1, void *arg2, void *arg3) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - fn(arg1, arg2, arg3); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_V(name) \ - extern "C" void *name() { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_W(name) \ - extern "C" void *name(void *arg) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_WW(name) \ - extern "C" void *name(void *arg1, void *arg2) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg1, arg2); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_WWW(name) \ - extern "C" void *name(void *arg1, void *arg2, void *arg3) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg1, arg2, arg3); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_WWWW(name) \ - extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg1, arg2, arg3, arg4); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_WWWWW(name) \ - extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4, \ - void *arg5) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg1, arg2, arg3, arg4, arg5); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); - -#define WRAP_W_WWWWWW(name) \ - extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4, \ - void *arg5, void *arg6) { \ - typedef decltype(name) *fntype; \ - static fntype fn = (fntype)getRealProcAddressOrDie(#name); \ - return fn(arg1, arg2, arg3, arg4, arg5, arg6); \ - } \ - INTERCEPT_WHEN_POSSIBLE(#name, name); -// }}} - -// ----------------- ASan own interface functions -------------------- -// Don't use the INTERFACE_FUNCTION machinery for this function as we actually -// want to call it in the __asan_init interceptor. -WRAP_W_V(__asan_should_detect_stack_use_after_return) -WRAP_W_V(__asan_get_shadow_memory_dynamic_address) - -extern "C" { - int __asan_option_detect_stack_use_after_return; - uptr __asan_shadow_memory_dynamic_address; - - // Manually wrap __asan_init as we need to initialize - // __asan_option_detect_stack_use_after_return afterwards. - void __asan_init() { - typedef void (*fntype)(); - static fntype fn = 0; - // __asan_init is expected to be called by only one thread. - if (fn) return; - - fn = (fntype)getRealProcAddressOrDie("__asan_init"); - fn(); - __asan_option_detect_stack_use_after_return = - (__asan_should_detect_stack_use_after_return() != 0); - __asan_shadow_memory_dynamic_address = - (uptr)__asan_get_shadow_memory_dynamic_address(); - InterceptHooks(); - } -} - -extern "C" void __asan_version_mismatch_check() { - // Do nothing. -} - -INTERFACE_FUNCTION(__asan_handle_no_return) -INTERFACE_FUNCTION(__asan_unhandled_exception_filter) - -INTERFACE_FUNCTION(__asan_report_store1) -INTERFACE_FUNCTION(__asan_report_store2) -INTERFACE_FUNCTION(__asan_report_store4) -INTERFACE_FUNCTION(__asan_report_store8) -INTERFACE_FUNCTION(__asan_report_store16) -INTERFACE_FUNCTION(__asan_report_store_n) - -INTERFACE_FUNCTION(__asan_report_load1) -INTERFACE_FUNCTION(__asan_report_load2) -INTERFACE_FUNCTION(__asan_report_load4) -INTERFACE_FUNCTION(__asan_report_load8) -INTERFACE_FUNCTION(__asan_report_load16) -INTERFACE_FUNCTION(__asan_report_load_n) - -INTERFACE_FUNCTION(__asan_store1) -INTERFACE_FUNCTION(__asan_store2) -INTERFACE_FUNCTION(__asan_store4) -INTERFACE_FUNCTION(__asan_store8) -INTERFACE_FUNCTION(__asan_store16) -INTERFACE_FUNCTION(__asan_storeN) - -INTERFACE_FUNCTION(__asan_load1) -INTERFACE_FUNCTION(__asan_load2) -INTERFACE_FUNCTION(__asan_load4) -INTERFACE_FUNCTION(__asan_load8) -INTERFACE_FUNCTION(__asan_load16) -INTERFACE_FUNCTION(__asan_loadN) - -INTERFACE_FUNCTION(__asan_memcpy); -INTERFACE_FUNCTION(__asan_memset); -INTERFACE_FUNCTION(__asan_memmove); - -INTERFACE_FUNCTION(__asan_set_shadow_00); -INTERFACE_FUNCTION(__asan_set_shadow_f1); -INTERFACE_FUNCTION(__asan_set_shadow_f2); -INTERFACE_FUNCTION(__asan_set_shadow_f3); -INTERFACE_FUNCTION(__asan_set_shadow_f5); -INTERFACE_FUNCTION(__asan_set_shadow_f8); - -INTERFACE_FUNCTION(__asan_alloca_poison); -INTERFACE_FUNCTION(__asan_allocas_unpoison); +// ASan own interface functions. +#define INTERFACE_FUNCTION(Name) INTERCEPT_SANITIZER_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "asan_interface.inc" -INTERFACE_FUNCTION(__asan_register_globals) -INTERFACE_FUNCTION(__asan_unregister_globals) +// Memory allocation functions. +INTERCEPT_WRAP_V_W(free) +INTERCEPT_WRAP_V_W(_free_base) +INTERCEPT_WRAP_V_WW(_free_dbg) -INTERFACE_FUNCTION(__asan_before_dynamic_init) -INTERFACE_FUNCTION(__asan_after_dynamic_init) +INTERCEPT_WRAP_W_W(malloc) +INTERCEPT_WRAP_W_W(_malloc_base) +INTERCEPT_WRAP_W_WWWW(_malloc_dbg) -INTERFACE_FUNCTION(__asan_poison_stack_memory) -INTERFACE_FUNCTION(__asan_unpoison_stack_memory) +INTERCEPT_WRAP_W_WW(calloc) +INTERCEPT_WRAP_W_WW(_calloc_base) +INTERCEPT_WRAP_W_WWWWW(_calloc_dbg) +INTERCEPT_WRAP_W_WWW(_calloc_impl) -INTERFACE_FUNCTION(__asan_poison_memory_region) -INTERFACE_FUNCTION(__asan_unpoison_memory_region) +INTERCEPT_WRAP_W_WW(realloc) +INTERCEPT_WRAP_W_WW(_realloc_base) +INTERCEPT_WRAP_W_WWW(_realloc_dbg) +INTERCEPT_WRAP_W_WWW(_recalloc) +INTERCEPT_WRAP_W_WWW(_recalloc_base) -INTERFACE_FUNCTION(__asan_address_is_poisoned) -INTERFACE_FUNCTION(__asan_region_is_poisoned) - -INTERFACE_FUNCTION(__asan_get_current_fake_stack) -INTERFACE_FUNCTION(__asan_addr_is_in_fake_stack) - -INTERFACE_FUNCTION(__asan_stack_malloc_0) -INTERFACE_FUNCTION(__asan_stack_malloc_1) -INTERFACE_FUNCTION(__asan_stack_malloc_2) -INTERFACE_FUNCTION(__asan_stack_malloc_3) -INTERFACE_FUNCTION(__asan_stack_malloc_4) -INTERFACE_FUNCTION(__asan_stack_malloc_5) -INTERFACE_FUNCTION(__asan_stack_malloc_6) -INTERFACE_FUNCTION(__asan_stack_malloc_7) -INTERFACE_FUNCTION(__asan_stack_malloc_8) -INTERFACE_FUNCTION(__asan_stack_malloc_9) -INTERFACE_FUNCTION(__asan_stack_malloc_10) - -INTERFACE_FUNCTION(__asan_stack_free_0) -INTERFACE_FUNCTION(__asan_stack_free_1) -INTERFACE_FUNCTION(__asan_stack_free_2) -INTERFACE_FUNCTION(__asan_stack_free_4) -INTERFACE_FUNCTION(__asan_stack_free_5) -INTERFACE_FUNCTION(__asan_stack_free_6) -INTERFACE_FUNCTION(__asan_stack_free_7) -INTERFACE_FUNCTION(__asan_stack_free_8) -INTERFACE_FUNCTION(__asan_stack_free_9) -INTERFACE_FUNCTION(__asan_stack_free_10) - -// FIXME: we might want to have a sanitizer_win_dll_thunk? -INTERFACE_FUNCTION(__sanitizer_annotate_contiguous_container) -INTERFACE_FUNCTION(__sanitizer_contiguous_container_find_bad_address) -INTERFACE_FUNCTION(__sanitizer_cov) -INTERFACE_FUNCTION(__sanitizer_cov_dump) -INTERFACE_FUNCTION(__sanitizer_dump_coverage) -INTERFACE_FUNCTION(__sanitizer_dump_trace_pc_guard_coverage) -INTERFACE_FUNCTION(__sanitizer_cov_indir_call16) -INTERFACE_FUNCTION(__sanitizer_cov_init) -INTERFACE_FUNCTION(__sanitizer_cov_module_init) -INTERFACE_FUNCTION(__sanitizer_cov_trace_basic_block) -INTERFACE_FUNCTION(__sanitizer_cov_trace_func_enter) -INTERFACE_FUNCTION(__sanitizer_cov_trace_pc_guard) -INTERFACE_FUNCTION(__sanitizer_cov_trace_pc_guard_init) -INTERFACE_FUNCTION(__sanitizer_cov_with_check) -INTERFACE_FUNCTION(__sanitizer_get_allocated_size) -INTERFACE_FUNCTION(__sanitizer_get_coverage_guards) -INTERFACE_FUNCTION(__sanitizer_get_current_allocated_bytes) -INTERFACE_FUNCTION(__sanitizer_get_estimated_allocated_size) -INTERFACE_FUNCTION(__sanitizer_get_free_bytes) -INTERFACE_FUNCTION(__sanitizer_get_heap_size) -INTERFACE_FUNCTION(__sanitizer_get_ownership) -INTERFACE_FUNCTION(__sanitizer_get_total_unique_caller_callee_pairs) -INTERFACE_FUNCTION(__sanitizer_get_total_unique_coverage) -INTERFACE_FUNCTION(__sanitizer_get_unmapped_bytes) -INTERFACE_FUNCTION(__sanitizer_maybe_open_cov_file) -INTERFACE_FUNCTION(__sanitizer_print_stack_trace) -INTERFACE_FUNCTION(__sanitizer_symbolize_pc) -INTERFACE_FUNCTION(__sanitizer_symbolize_global) -INTERFACE_FUNCTION(__sanitizer_ptr_cmp) -INTERFACE_FUNCTION(__sanitizer_ptr_sub) -INTERFACE_FUNCTION(__sanitizer_report_error_summary) -INTERFACE_FUNCTION(__sanitizer_reset_coverage) -INTERFACE_FUNCTION(__sanitizer_get_number_of_counters) -INTERFACE_FUNCTION(__sanitizer_update_counter_bitset_and_clear_counters) -INTERFACE_FUNCTION(__sanitizer_sandbox_on_notify) -INTERFACE_FUNCTION(__sanitizer_set_death_callback) -INTERFACE_FUNCTION(__sanitizer_set_report_path) -INTERFACE_FUNCTION(__sanitizer_set_report_fd) -INTERFACE_FUNCTION(__sanitizer_unaligned_load16) -INTERFACE_FUNCTION(__sanitizer_unaligned_load32) -INTERFACE_FUNCTION(__sanitizer_unaligned_load64) -INTERFACE_FUNCTION(__sanitizer_unaligned_store16) -INTERFACE_FUNCTION(__sanitizer_unaligned_store32) -INTERFACE_FUNCTION(__sanitizer_unaligned_store64) -INTERFACE_FUNCTION(__sanitizer_verify_contiguous_container) -INTERFACE_FUNCTION(__sanitizer_install_malloc_and_free_hooks) -INTERFACE_FUNCTION(__sanitizer_start_switch_fiber) -INTERFACE_FUNCTION(__sanitizer_finish_switch_fiber) -INTERFACE_FUNCTION(__sanitizer_get_module_and_offset_for_pc) - -// TODO(timurrrr): Add more interface functions on the as-needed basis. - -// ----------------- Memory allocation functions --------------------- -WRAP_V_W(free) -WRAP_V_W(_free_base) -WRAP_V_WW(_free_dbg) - -WRAP_W_W(malloc) -WRAP_W_W(_malloc_base) -WRAP_W_WWWW(_malloc_dbg) - -WRAP_W_WW(calloc) -WRAP_W_WW(_calloc_base) -WRAP_W_WWWWW(_calloc_dbg) -WRAP_W_WWW(_calloc_impl) - -WRAP_W_WW(realloc) -WRAP_W_WW(_realloc_base) -WRAP_W_WWW(_realloc_dbg) -WRAP_W_WWW(_recalloc) -WRAP_W_WWW(_recalloc_base) - -WRAP_W_W(_msize) -WRAP_W_W(_expand) -WRAP_W_W(_expand_dbg) +INTERCEPT_WRAP_W_W(_msize) +INTERCEPT_WRAP_W_W(_expand) +INTERCEPT_WRAP_W_W(_expand_dbg) // TODO(timurrrr): Might want to add support for _aligned_* allocation // functions to detect a bit more bugs. Those functions seem to wrap malloc(). @@ -405,20 +58,6 @@ WRAP_W_W(_expand_dbg) INTERCEPT_LIBRARY_FUNCTION(atoi); INTERCEPT_LIBRARY_FUNCTION(atol); - -#ifdef _WIN64 -INTERCEPT_LIBRARY_FUNCTION(__C_specific_handler); -#else -INTERCEPT_LIBRARY_FUNCTION(_except_handler3); - -// _except_handler4 checks -GS cookie which is different for each module, so we -// can't use INTERCEPT_LIBRARY_FUNCTION(_except_handler4). -INTERCEPTOR(int, _except_handler4, void *a, void *b, void *c, void *d) { - __asan_handle_no_return(); - return REAL(_except_handler4)(a, b, c, d); -} -#endif - INTERCEPT_LIBRARY_FUNCTION(frexp); INTERCEPT_LIBRARY_FUNCTION(longjmp); #if SANITIZER_INTERCEPT_MEMCHR @@ -443,41 +82,71 @@ INTERCEPT_LIBRARY_FUNCTION(strpbrk); INTERCEPT_LIBRARY_FUNCTION(strrchr); INTERCEPT_LIBRARY_FUNCTION(strspn); INTERCEPT_LIBRARY_FUNCTION(strstr); +INTERCEPT_LIBRARY_FUNCTION(strtok); INTERCEPT_LIBRARY_FUNCTION(strtol); INTERCEPT_LIBRARY_FUNCTION(wcslen); +INTERCEPT_LIBRARY_FUNCTION(wcsnlen); -// Must be after all the interceptor declarations due to the way INTERCEPT_HOOKS -// is defined. -void InterceptHooks() { - INTERCEPT_HOOKS(); -#ifndef _WIN64 - INTERCEPT_FUNCTION(_except_handler4); -#endif +#ifdef _WIN64 +INTERCEPT_LIBRARY_FUNCTION(__C_specific_handler); +#else +INTERCEPT_LIBRARY_FUNCTION(_except_handler3); +// _except_handler4 checks -GS cookie which is different for each module, so we +// can't use INTERCEPT_LIBRARY_FUNCTION(_except_handler4). +INTERCEPTOR(int, _except_handler4, void *a, void *b, void *c, void *d) { + __asan_handle_no_return(); + return REAL(_except_handler4)(a, b, c, d); } +#endif -// We want to call __asan_init before C/C++ initializers/constructors are -// executed, otherwise functions like memset might be invoked. -// For some strange reason, merely linking in asan_preinit.cc doesn't work -// as the callback is never called... Is link.exe doing something too smart? +// Window specific functions not included in asan_interface.inc. +INTERCEPT_WRAP_W_V(__asan_should_detect_stack_use_after_return) +INTERCEPT_WRAP_W_V(__asan_get_shadow_memory_dynamic_address) +INTERCEPT_WRAP_W_W(__asan_unhandled_exception_filter) -// In DLLs, the callbacks are expected to return 0, -// otherwise CRT initialization fails. -static int call_asan_init() { - __asan_init(); +using namespace __sanitizer; + +extern "C" { +int __asan_option_detect_stack_use_after_return; +uptr __asan_shadow_memory_dynamic_address; +} // extern "C" + +static int asan_dll_thunk_init() { + typedef void (*fntype)(); + static fntype fn = 0; + // asan_dll_thunk_init is expected to be called by only one thread. + if (fn) return 0; + + // Ensure all interception was executed. + __dll_thunk_init(); + + fn = (fntype) dllThunkGetRealAddrOrDie("__asan_init"); + fn(); + __asan_option_detect_stack_use_after_return = + (__asan_should_detect_stack_use_after_return() != 0); + __asan_shadow_memory_dynamic_address = + (uptr)__asan_get_shadow_memory_dynamic_address(); + +#ifndef _WIN64 + INTERCEPT_FUNCTION(_except_handler4); +#endif + // In DLLs, the callbacks are expected to return 0, + // otherwise CRT initialization fails. return 0; } + #pragma section(".CRT$XIB", long, read) // NOLINT -__declspec(allocate(".CRT$XIB")) int (*__asan_preinit)() = call_asan_init; +__declspec(allocate(".CRT$XIB")) int (*__asan_preinit)() = asan_dll_thunk_init; static void WINAPI asan_thread_init(void *mod, unsigned long reason, - void *reserved) { - if (reason == /*DLL_PROCESS_ATTACH=*/1) __asan_init(); + void *reserved) { + if (reason == /*DLL_PROCESS_ATTACH=*/1) asan_dll_thunk_init(); } #pragma section(".CRT$XLAB", long, read) // NOLINT __declspec(allocate(".CRT$XLAB")) void (WINAPI *__asan_tls_init)(void *, unsigned long, void *) = asan_thread_init; -ASAN_LINK_GLOBALS_WIN() +WIN_FORCE_LINK(__asan_dso_reg_hook) -#endif // ASAN_DLL_THUNK +#endif // SANITIZER_DLL_THUNK diff --git a/contrib/compiler-rt/lib/asan/asan_win_dynamic_runtime_thunk.cc b/contrib/compiler-rt/lib/asan/asan_win_dynamic_runtime_thunk.cc index 8e42f03..416c73b 100644 --- a/contrib/compiler-rt/lib/asan/asan_win_dynamic_runtime_thunk.cc +++ b/contrib/compiler-rt/lib/asan/asan_win_dynamic_runtime_thunk.cc @@ -14,20 +14,24 @@ // using the default "import library" generated when linking the DLL RTL. // // This includes: +// - creating weak aliases to default implementation imported from asan dll. // - forwarding the detect_stack_use_after_return runtime option // - working around deficiencies of the MD runtime // - installing a custom SEH handler // //===----------------------------------------------------------------------===// -// Only compile this code when building asan_dynamic_runtime_thunk.lib -// Using #ifdef rather than relying on Makefiles etc. -// simplifies the build procedure. -#ifdef ASAN_DYNAMIC_RUNTIME_THUNK -#include "asan_globals_win.h" +#ifdef SANITIZER_DYNAMIC_RUNTIME_THUNK +#define SANITIZER_IMPORT_INTERFACE 1 +#include "sanitizer_common/sanitizer_win_defs.h" #define WIN32_LEAN_AND_MEAN #include <windows.h> +// Define weak alias for all weak functions imported from asan dll. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) WIN_WEAK_IMPORT_DEF(Name) +#include "asan_interface.inc" + // First, declare CRT sections we'll be using in this file #pragma section(".CRT$XIB", long, read) // NOLINT #pragma section(".CRT$XID", long, read) // NOLINT @@ -122,6 +126,6 @@ __declspec(allocate(".CRT$XCAB")) int (*__asan_seh_interceptor)() = SetSEHFilter; } -ASAN_LINK_GLOBALS_WIN() +WIN_FORCE_LINK(__asan_dso_reg_hook) -#endif // ASAN_DYNAMIC_RUNTIME_THUNK +#endif // SANITIZER_DYNAMIC_RUNTIME_THUNK diff --git a/contrib/compiler-rt/lib/asan/asan_win_weak_interception.cc b/contrib/compiler-rt/lib/asan/asan_win_weak_interception.cc new file mode 100644 index 0000000..ca26f91 --- /dev/null +++ b/contrib/compiler-rt/lib/asan/asan_win_weak_interception.cc @@ -0,0 +1,23 @@ +//===-- asan_win_weak_interception.cc -------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This module should be included in Address Sanitizer when it is implemented as +// a shared library on Windows (dll), in order to delegate the calls of weak +// functions to the implementation in the main executable when a strong +// definition is provided. +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC +#include "sanitizer_common/sanitizer_win_weak_interception.h" +#include "asan_interface_internal.h" +// Check if strong definitions for weak functions are present in the main +// executable. If that is the case, override dll functions to point to strong +// implementations. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "asan_interface.inc" +#endif // SANITIZER_DYNAMIC diff --git a/contrib/compiler-rt/lib/asan/weak_symbols.txt b/contrib/compiler-rt/lib/asan/weak_symbols.txt index ba7b027..fe680f8 100644 --- a/contrib/compiler-rt/lib/asan/weak_symbols.txt +++ b/contrib/compiler-rt/lib/asan/weak_symbols.txt @@ -1,3 +1,12 @@ ___asan_default_options ___asan_default_suppressions ___asan_on_error +___asan_set_shadow_00 +___asan_set_shadow_f1 +___asan_set_shadow_f2 +___asan_set_shadow_f3 +___asan_set_shadow_f4 +___asan_set_shadow_f5 +___asan_set_shadow_f6 +___asan_set_shadow_f7 +___asan_set_shadow_f8 diff --git a/contrib/compiler-rt/lib/builtins/README.txt b/contrib/compiler-rt/lib/builtins/README.txt index ad36e4e..e603dfa 100644 --- a/contrib/compiler-rt/lib/builtins/README.txt +++ b/contrib/compiler-rt/lib/builtins/README.txt @@ -45,6 +45,7 @@ si_int __ctzsi2(si_int a); // count trailing zeros si_int __ctzdi2(di_int a); // count trailing zeros si_int __ctzti2(ti_int a); // count trailing zeros +si_int __ffssi2(si_int a); // find least significant 1 bit si_int __ffsdi2(di_int a); // find least significant 1 bit si_int __ffsti2(ti_int a); // find least significant 1 bit @@ -56,8 +57,8 @@ si_int __popcountsi2(si_int a); // bit population si_int __popcountdi2(di_int a); // bit population si_int __popcountti2(ti_int a); // bit population -uint32_t __bswapsi2(uint32_t a); // a byteswapped, arm only -uint64_t __bswapdi2(uint64_t a); // a byteswapped, arm only +uint32_t __bswapsi2(uint32_t a); // a byteswapped +uint64_t __bswapdi2(uint64_t a); // a byteswapped // Integral arithmetic diff --git a/contrib/compiler-rt/lib/builtins/adddf3.c b/contrib/compiler-rt/lib/builtins/adddf3.c index 8b7aae0..c528e9e 100644 --- a/contrib/compiler-rt/lib/builtins/adddf3.c +++ b/contrib/compiler-rt/lib/builtins/adddf3.c @@ -15,8 +15,13 @@ #define DOUBLE_PRECISION #include "fp_add_impl.inc" -ARM_EABI_FNALIAS(dadd, adddf3) - COMPILER_RT_ABI double __adddf3(double a, double b){ return __addXf3__(a, b); } + +#if defined(__ARM_EABI__) +AEABI_RTABI double __aeabi_dadd(double a, double b) { + return __adddf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/addsf3.c b/contrib/compiler-rt/lib/builtins/addsf3.c index 0f5d6ea..fe57068 100644 --- a/contrib/compiler-rt/lib/builtins/addsf3.c +++ b/contrib/compiler-rt/lib/builtins/addsf3.c @@ -15,8 +15,13 @@ #define SINGLE_PRECISION #include "fp_add_impl.inc" -ARM_EABI_FNALIAS(fadd, addsf3) - COMPILER_RT_ABI float __addsf3(float a, float b) { return __addXf3__(a, b); } + +#if defined(__ARM_EABI__) +AEABI_RTABI float __aeabi_fadd(float a, float b) { + return __addsf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/arm/addsf3.S b/contrib/compiler-rt/lib/builtins/arm/addsf3.S new file mode 100644 index 0000000..362b5c1 --- /dev/null +++ b/contrib/compiler-rt/lib/builtins/arm/addsf3.S @@ -0,0 +1,277 @@ +/*===-- addsf3.S - Adds two single precision floating pointer numbers-----===// + * + * The LLVM Compiler Infrastructure + * + * This file is dual licensed under the MIT and the University of Illinois Open + * Source Licenses. See LICENSE.TXT for details. + * + *===----------------------------------------------------------------------===// + * + * This file implements the __addsf3 (single precision floating pointer number + * addition with the IEEE-754 default rounding (to nearest, ties to even) + * function for the ARM Thumb1 ISA. + * + *===----------------------------------------------------------------------===*/ + +#include "../assembly.h" +#define significandBits 23 +#define typeWidth 32 + + .syntax unified + .text + .thumb + .p2align 2 + +DEFINE_AEABI_FUNCTION_ALIAS(__aeabi_fadd, __addsf3) + +DEFINE_COMPILERRT_THUMB_FUNCTION(__addsf3) + push {r4, r5, r6, r7, lr} + // Get the absolute value of a and b. + lsls r2, r0, #1 + lsls r3, r1, #1 + lsrs r2, r2, #1 /* aAbs */ + beq LOCAL_LABEL(a_zero_nan_inf) + lsrs r3, r3, #1 /* bAbs */ + beq LOCAL_LABEL(zero_nan_inf) + + // Detect if a or b is infinity or Nan. + lsrs r6, r2, #(significandBits) + lsrs r7, r3, #(significandBits) + cmp r6, #0xFF + beq LOCAL_LABEL(zero_nan_inf) + cmp r7, #0xFF + beq LOCAL_LABEL(zero_nan_inf) + + // Swap Rep and Abs so that a and aAbs has the larger absolute value. + cmp r2, r3 + bhs LOCAL_LABEL(no_swap) + movs r4, r0 + movs r5, r2 + movs r0, r1 + movs r2, r3 + movs r1, r4 + movs r3, r5 +LOCAL_LABEL(no_swap): + + // Get the significands and shift them to give us round, guard and sticky. + lsls r4, r0, #(typeWidth - significandBits) + lsrs r4, r4, #(typeWidth - significandBits - 3) /* aSignificand << 3 */ + lsls r5, r1, #(typeWidth - significandBits) + lsrs r5, r5, #(typeWidth - significandBits - 3) /* bSignificand << 3 */ + + // Get the implicitBit. + movs r6, #1 + lsls r6, r6, #(significandBits + 3) + + // Get aExponent and set implicit bit if necessary. + lsrs r2, r2, #(significandBits) + beq LOCAL_LABEL(a_done_implicit_bit) + orrs r4, r6 +LOCAL_LABEL(a_done_implicit_bit): + + // Get bExponent and set implicit bit if necessary. + lsrs r3, r3, #(significandBits) + beq LOCAL_LABEL(b_done_implicit_bit) + orrs r5, r6 +LOCAL_LABEL(b_done_implicit_bit): + + // Get the difference in exponents. + subs r6, r2, r3 + beq LOCAL_LABEL(done_align) + + // If b is denormal, then a must be normal as align > 0, and we only need to + // right shift bSignificand by (align - 1) bits. + cmp r3, #0 + bne 1f + subs r6, r6, #1 +1: + + // No longer needs bExponent. r3 is dead here. + // Set sticky bits of b: sticky = bSignificand << (typeWidth - align). + movs r3, #(typeWidth) + subs r3, r3, r6 + movs r7, r5 + lsls r7, r3 + beq 1f + movs r7, #1 +1: + + // bSignificand = bSignificand >> align | sticky; + lsrs r5, r6 + orrs r5, r7 + bne LOCAL_LABEL(done_align) + movs r5, #1 // sticky; b is known to be non-zero. + +LOCAL_LABEL(done_align): + // isSubtraction = (aRep ^ bRep) >> 31; + movs r7, r0 + eors r7, r1 + lsrs r7, #31 + bne LOCAL_LABEL(do_substraction) + + // Same sign, do Addition. + + // aSignificand += bSignificand; + adds r4, r4, r5 + + // Check carry bit. + movs r6, #1 + lsls r6, r6, #(significandBits + 3 + 1) + movs r7, r4 + ands r7, r6 + beq LOCAL_LABEL(form_result) + // If the addition carried up, we need to right-shift the result and + // adjust the exponent. + movs r7, r4 + movs r6, #1 + ands r7, r6 // sticky = aSignificand & 1; + lsrs r4, #1 + orrs r4, r7 // result Significand + adds r2, #1 // result Exponent + // If we have overflowed the type, return +/- infinity. + cmp r2, 0xFF + beq LOCAL_LABEL(ret_inf) + +LOCAL_LABEL(form_result): + // Shift the sign, exponent and significand into place. + lsrs r0, #(typeWidth - 1) + lsls r0, #(typeWidth - 1) // Get Sign. + lsls r2, #(significandBits) + orrs r0, r2 + movs r1, r4 + lsls r4, #(typeWidth - significandBits - 3) + lsrs r4, #(typeWidth - significandBits) + orrs r0, r4 + + // Final rounding. The result may overflow to infinity, but that is the + // correct result in that case. + // roundGuardSticky = aSignificand & 0x7; + movs r2, #0x7 + ands r1, r2 + // if (roundGuardSticky > 0x4) result++; + + cmp r1, #0x4 + blt LOCAL_LABEL(done_round) + beq 1f + adds r0, #1 + pop {r4, r5, r6, r7, pc} +1: + + // if (roundGuardSticky == 0x4) result += result & 1; + movs r1, r0 + lsrs r1, #1 + bcc LOCAL_LABEL(done_round) + adds r0, r0, #1 +LOCAL_LABEL(done_round): + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(do_substraction): + subs r4, r4, r5 // aSignificand -= bSignificand; + beq LOCAL_LABEL(ret_zero) + movs r6, r4 + cmp r2, 0 + beq LOCAL_LABEL(form_result) // if a's exp is 0, no need to normalize. + // If partial cancellation occured, we need to left-shift the result + // and adjust the exponent: + lsrs r6, r6, #(significandBits + 3) + bne LOCAL_LABEL(form_result) + + push {r0, r1, r2, r3} + movs r0, r4 + bl __clzsi2 + movs r5, r0 + pop {r0, r1, r2, r3} + // shift = rep_clz(aSignificand) - rep_clz(implicitBit << 3); + subs r5, r5, #(typeWidth - significandBits - 3 - 1) + // aSignificand <<= shift; aExponent -= shift; + lsls r4, r5 + subs r2, r2, r5 + bgt LOCAL_LABEL(form_result) + + // Do normalization if aExponent <= 0. + movs r6, #1 + subs r6, r6, r2 // 1 - aExponent; + movs r2, #0 // aExponent = 0; + movs r3, #(typeWidth) // bExponent is dead. + subs r3, r3, r6 + movs r7, r4 + lsls r7, r3 // stickyBit = (bool)(aSignificant << (typeWidth - align)) + beq 1f + movs r7, #1 +1: + lsrs r4, r6 /* aSignificand >> shift */ + orrs r4, r7 + b LOCAL_LABEL(form_result) + +LOCAL_LABEL(ret_zero): + movs r0, #0 + pop {r4, r5, r6, r7, pc} + + +LOCAL_LABEL(a_zero_nan_inf): + lsrs r3, r3, #1 + +LOCAL_LABEL(zero_nan_inf): + // Here r2 has aAbs, r3 has bAbs + movs r4, #0xFF + lsls r4, r4, #(significandBits) // Make +inf. + + cmp r2, r4 + bhi LOCAL_LABEL(a_is_nan) + cmp r3, r4 + bhi LOCAL_LABEL(b_is_nan) + + cmp r2, r4 + bne LOCAL_LABEL(a_is_rational) + // aAbs is INF. + eors r1, r0 // aRep ^ bRep. + movs r6, #1 + lsls r6, r6, #(typeWidth - 1) // get sign mask. + cmp r1, r6 // if they only differ on sign bit, it's -INF + INF + beq LOCAL_LABEL(a_is_nan) + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(a_is_rational): + cmp r3, r4 + bne LOCAL_LABEL(b_is_rational) + movs r0, r1 + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(b_is_rational): + // either a or b or both are zero. + adds r4, r2, r3 + beq LOCAL_LABEL(both_zero) + cmp r2, #0 // is absA 0 ? + beq LOCAL_LABEL(ret_b) + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(both_zero): + ands r0, r1 // +0 + -0 = +0 + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(ret_b): + movs r0, r1 + +LOCAL_LABEL(ret): + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(b_is_nan): + movs r0, r1 +LOCAL_LABEL(a_is_nan): + movs r1, #1 + lsls r1, r1, #(significandBits -1) // r1 is quiet bit. + orrs r0, r1 + pop {r4, r5, r6, r7, pc} + +LOCAL_LABEL(ret_inf): + movs r4, #0xFF + lsls r4, r4, #(significandBits) + orrs r0, r4 + lsrs r0, r0, #(significandBits) + lsls r0, r0, #(significandBits) + pop {r4, r5, r6, r7, pc} + + +END_COMPILERRT_FUNCTION(__addsf3) + +NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmp.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmp.S index 8008f5f..3e7a8b8 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmp.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmp.S @@ -30,13 +30,32 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_cdcmpeq) push {r0-r3, lr} bl __aeabi_cdcmpeq_check_nan cmp r0, #1 +#if __ARM_ARCH_ISA_THUMB == 1 + beq 1f + // NaN has been ruled out, so __aeabi_cdcmple can't trap + mov r0, sp + ldm r0, {r0-r3} + bl __aeabi_cdcmple + pop {r0-r3, pc} +1: + // Z = 0, C = 1 + movs r0, #0xF + lsls r0, r0, #31 + pop {r0-r3, pc} +#else pop {r0-r3, lr} // NaN has been ruled out, so __aeabi_cdcmple can't trap bne __aeabi_cdcmple +#if defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7EM__) + mov ip, #APSR_C + msr APSR_nzcvq, ip +#else msr CPSR_f, #APSR_C +#endif JMP(lr) +#endif END_COMPILERRT_FUNCTION(__aeabi_cdcmpeq) @@ -59,19 +78,48 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_cdcmple) bl __aeabi_dcmplt cmp r0, #1 +#if __ARM_ARCH_ISA_THUMB == 1 + bne 1f + // Z = 0, C = 0 + movs r0, #1 + lsls r0, r0, #1 + pop {r0-r3, pc} +1: + mov r0, sp + ldm r0, {r0-r3} + bl __aeabi_dcmpeq + cmp r0, #1 + bne 2f + // Z = 1, C = 1 + movs r0, #2 + lsls r0, r0, #31 + pop {r0-r3, pc} +2: + // Z = 0, C = 1 + movs r0, #0xF + lsls r0, r0, #31 + pop {r0-r3, pc} +#else + ITT(eq) moveq ip, #0 beq 1f ldm sp, {r0-r3} bl __aeabi_dcmpeq cmp r0, #1 + ITE(eq) moveq ip, #(APSR_C | APSR_Z) movne ip, #(APSR_C) 1: +#if defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7EM__) + msr APSR_nzcvq, ip +#else msr CPSR_f, ip +#endif pop {r0-r3} POP_PC() +#endif END_COMPILERRT_FUNCTION(__aeabi_cdcmple) // int __aeabi_cdrcmple(double a, double b) { diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmpeq_check_nan.c b/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmpeq_check_nan.c index 577f6b2c..7578433 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmpeq_check_nan.c +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_cdcmpeq_check_nan.c @@ -8,9 +8,9 @@ //===----------------------------------------------------------------------===// #include <stdint.h> +#include "../int_lib.h" -__attribute__((pcs("aapcs"))) -__attribute__((visibility("hidden"))) +AEABI_RTABI __attribute__((visibility("hidden"))) int __aeabi_cdcmpeq_check_nan(double a, double b) { return __builtin_isnan(a) || __builtin_isnan(b); } diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmp.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmp.S index 274baf7a..1f304ff 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmp.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmp.S @@ -30,13 +30,32 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_cfcmpeq) push {r0-r3, lr} bl __aeabi_cfcmpeq_check_nan cmp r0, #1 +#if __ARM_ARCH_ISA_THUMB == 1 + beq 1f + // NaN has been ruled out, so __aeabi_cfcmple can't trap + mov r0, sp + ldm r0, {r0-r3} + bl __aeabi_cfcmple + pop {r0-r3, pc} +1: + // Z = 0, C = 1 + movs r0, #0xF + lsls r0, r0, #31 + pop {r0-r3, pc} +#else pop {r0-r3, lr} // NaN has been ruled out, so __aeabi_cfcmple can't trap bne __aeabi_cfcmple +#if defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7EM__) + mov ip, #APSR_C + msr APSR_nzcvq, ip +#else msr CPSR_f, #APSR_C +#endif JMP(lr) +#endif END_COMPILERRT_FUNCTION(__aeabi_cfcmpeq) @@ -59,19 +78,48 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_cfcmple) bl __aeabi_fcmplt cmp r0, #1 +#if __ARM_ARCH_ISA_THUMB == 1 + bne 1f + // Z = 0, C = 0 + movs r0, #1 + lsls r0, r0, #1 + pop {r0-r3, pc} +1: + mov r0, sp + ldm r0, {r0-r3} + bl __aeabi_fcmpeq + cmp r0, #1 + bne 2f + // Z = 1, C = 1 + movs r0, #2 + lsls r0, r0, #31 + pop {r0-r3, pc} +2: + // Z = 0, C = 1 + movs r0, #0xF + lsls r0, r0, #31 + pop {r0-r3, pc} +#else + ITT(eq) moveq ip, #0 beq 1f ldm sp, {r0-r3} bl __aeabi_fcmpeq cmp r0, #1 + ITE(eq) moveq ip, #(APSR_C | APSR_Z) movne ip, #(APSR_C) 1: +#if defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7EM__) + msr APSR_nzcvq, ip +#else msr CPSR_f, ip +#endif pop {r0-r3} POP_PC() +#endif END_COMPILERRT_FUNCTION(__aeabi_cfcmple) // int __aeabi_cfrcmple(float a, float b) { diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmpeq_check_nan.c b/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmpeq_check_nan.c index 992e31f..43dde9a 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmpeq_check_nan.c +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_cfcmpeq_check_nan.c @@ -8,9 +8,9 @@ //===----------------------------------------------------------------------===// #include <stdint.h> +#include "../int_lib.h" -__attribute__((pcs("aapcs"))) -__attribute__((visibility("hidden"))) +AEABI_RTABI __attribute__((visibility("hidden"))) int __aeabi_cfcmpeq_check_nan(float a, float b) { return __builtin_isnan(a) || __builtin_isnan(b); } diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_dcmp.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_dcmp.S index 43e4392..9fa78b4 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_dcmp.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_dcmp.S @@ -18,18 +18,27 @@ // } // } +#if defined(COMPILER_RT_ARMHF_TARGET) +# define CONVERT_DCMP_ARGS_TO_DF2_ARGS \ + vmov d0, r0, r1 SEPARATOR \ + vmov d1, r2, r3 +#else +# define CONVERT_DCMP_ARGS_TO_DF2_ARGS +#endif + #define DEFINE_AEABI_DCMP(cond) \ .syntax unified SEPARATOR \ .p2align 2 SEPARATOR \ DEFINE_COMPILERRT_FUNCTION(__aeabi_dcmp ## cond) \ push { r4, lr } SEPARATOR \ + CONVERT_DCMP_ARGS_TO_DF2_ARGS SEPARATOR \ bl SYMBOL_NAME(__ ## cond ## df2) SEPARATOR \ cmp r0, #0 SEPARATOR \ b ## cond 1f SEPARATOR \ - mov r0, #0 SEPARATOR \ + movs r0, #0 SEPARATOR \ pop { r4, pc } SEPARATOR \ 1: SEPARATOR \ - mov r0, #1 SEPARATOR \ + movs r0, #1 SEPARATOR \ pop { r4, pc } SEPARATOR \ END_COMPILERRT_FUNCTION(__aeabi_dcmp ## cond) diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_div0.c b/contrib/compiler-rt/lib/builtins/arm/aeabi_div0.c index ccc95fa..dc30313 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_div0.c +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_div0.c @@ -26,16 +26,18 @@ * line. */ +#include "../int_lib.h" + /* provide an unused declaration to pacify pendantic compilation */ extern unsigned char declaration; #if defined(__ARM_EABI__) -int __attribute__((weak)) __attribute__((visibility("hidden"))) +AEABI_RTABI int __attribute__((weak)) __attribute__((visibility("hidden"))) __aeabi_idiv0(int return_value) { return return_value; } -long long __attribute__((weak)) __attribute__((visibility("hidden"))) +AEABI_RTABI long long __attribute__((weak)) __attribute__((visibility("hidden"))) __aeabi_ldiv0(long long return_value) { return return_value; } diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_drsub.c b/contrib/compiler-rt/lib/builtins/arm/aeabi_drsub.c index fc17d5a..1254886 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_drsub.c +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_drsub.c @@ -10,10 +10,10 @@ #define DOUBLE_PRECISION #include "../fp_lib.h" -COMPILER_RT_ABI fp_t +AEABI_RTABI fp_t __aeabi_dsub(fp_t, fp_t); -COMPILER_RT_ABI fp_t +AEABI_RTABI fp_t __aeabi_drsub(fp_t a, fp_t b) { return __aeabi_dsub(b, a); } diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_fcmp.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_fcmp.S index 8e7774b..ea5b96c 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_fcmp.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_fcmp.S @@ -18,11 +18,20 @@ // } // } +#if defined(COMPILER_RT_ARMHF_TARGET) +# define CONVERT_FCMP_ARGS_TO_SF2_ARGS \ + vmov s0, r0 SEPARATOR \ + vmov s1, r1 +#else +# define CONVERT_FCMP_ARGS_TO_SF2_ARGS +#endif + #define DEFINE_AEABI_FCMP(cond) \ .syntax unified SEPARATOR \ .p2align 2 SEPARATOR \ DEFINE_COMPILERRT_FUNCTION(__aeabi_fcmp ## cond) \ push { r4, lr } SEPARATOR \ + CONVERT_FCMP_ARGS_TO_SF2_ARGS SEPARATOR \ bl SYMBOL_NAME(__ ## cond ## sf2) SEPARATOR \ cmp r0, #0 SEPARATOR \ b ## cond 1f SEPARATOR \ diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_frsub.c b/contrib/compiler-rt/lib/builtins/arm/aeabi_frsub.c index 64258dc..34f2303 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_frsub.c +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_frsub.c @@ -10,10 +10,10 @@ #define SINGLE_PRECISION #include "../fp_lib.h" -COMPILER_RT_ABI fp_t +AEABI_RTABI fp_t __aeabi_fsub(fp_t, fp_t); -COMPILER_RT_ABI fp_t +AEABI_RTABI fp_t __aeabi_frsub(fp_t a, fp_t b) { return __aeabi_fsub(b, a); } diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_idivmod.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_idivmod.S index 6befc16..4419929 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_idivmod.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_idivmod.S @@ -35,7 +35,7 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_idivmod) push {r0, r1, lr} bl SYMBOL_NAME(__divsi3) pop {r1, r2, r3} // now r0 = quot, r1 = num, r2 = denom - muls r2, r2, r0 // r2 = quot * denom + muls r2, r0, r2 // r2 = quot * denom subs r1, r1, r2 JMP (r3) #else // defined(USE_THUMB_1) diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_ldivmod.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_ldivmod.S index 3dae14e..038ae5d 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_ldivmod.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_ldivmod.S @@ -23,23 +23,23 @@ .syntax unified .p2align 2 DEFINE_COMPILERRT_FUNCTION(__aeabi_ldivmod) - push {r11, lr} + push {r6, lr} sub sp, sp, #16 - add r12, sp, #8 - str r12, [sp] + add r6, sp, #8 + str r6, [sp] #if defined(__MINGW32__) - mov r12, r0 - mov r0, r2 - mov r2, r12 - mov r12, r1 - mov r1, r3 - mov r3, r12 + movs r6, r0 + movs r0, r2 + movs r2, r6 + movs r6, r1 + movs r1, r3 + movs r3, r6 #endif bl SYMBOL_NAME(__divmoddi4) ldr r2, [sp, #8] ldr r3, [sp, #12] add sp, sp, #16 - pop {r11, pc} + pop {r6, pc} END_COMPILERRT_FUNCTION(__aeabi_ldivmod) NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_memset.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_memset.S index 48edd89..633f592 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_memset.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_memset.S @@ -26,7 +26,7 @@ DEFINE_AEABI_FUNCTION_ALIAS(__aeabi_memset8, __aeabi_memset) DEFINE_COMPILERRT_FUNCTION(__aeabi_memclr) mov r2, r1 - mov r1, #0 + movs r1, #0 b memset END_COMPILERRT_FUNCTION(__aeabi_memclr) diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_uidivmod.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_uidivmod.S index 0330f33..37dae4a 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_uidivmod.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_uidivmod.S @@ -37,7 +37,7 @@ DEFINE_COMPILERRT_FUNCTION(__aeabi_uidivmod) push {r0, r1, lr} bl SYMBOL_NAME(__aeabi_uidiv) pop {r1, r2, r3} - muls r2, r2, r0 // r2 = quot * denom + muls r2, r0, r2 // r2 = quot * denom subs r1, r1, r2 JMP (r3) LOCAL_LABEL(case_denom_larger): diff --git a/contrib/compiler-rt/lib/builtins/arm/aeabi_uldivmod.S b/contrib/compiler-rt/lib/builtins/arm/aeabi_uldivmod.S index bc26e56..be343b6 100644 --- a/contrib/compiler-rt/lib/builtins/arm/aeabi_uldivmod.S +++ b/contrib/compiler-rt/lib/builtins/arm/aeabi_uldivmod.S @@ -23,23 +23,23 @@ .syntax unified .p2align 2 DEFINE_COMPILERRT_FUNCTION(__aeabi_uldivmod) - push {r11, lr} + push {r6, lr} sub sp, sp, #16 - add r12, sp, #8 - str r12, [sp] + add r6, sp, #8 + str r6, [sp] #if defined(__MINGW32__) - mov r12, r0 - mov r0, r2 - mov r2, r12 - mov r12, r1 - mov r1, r3 - mov r3, r12 + movs r6, r0 + movs r0, r2 + movs r2, r6 + movs r6, r1 + movs r1, r3 + movs r3, r6 #endif bl SYMBOL_NAME(__udivmoddi4) ldr r2, [sp, #8] ldr r3, [sp, #12] add sp, sp, #16 - pop {r11, pc} + pop {r6, pc} END_COMPILERRT_FUNCTION(__aeabi_uldivmod) NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/builtins/arm/comparesf2.S b/contrib/compiler-rt/lib/builtins/arm/comparesf2.S index 74ff0d1..1f7031c 100644 --- a/contrib/compiler-rt/lib/builtins/arm/comparesf2.S +++ b/contrib/compiler-rt/lib/builtins/arm/comparesf2.S @@ -74,7 +74,7 @@ DEFINE_COMPILERRT_FUNCTION(__eqsf2) // the subsequent operations. #if defined(USE_THUMB_1) lsrs r6, r3, #1 - orrs r6, r2, r6 + orrs r6, r2 #else orrs r12, r2, r3, lsr #1 #endif @@ -203,7 +203,7 @@ DEFINE_COMPILERRT_FUNCTION(__gtsf2) lsls r2, r0, #1 lsls r3, r1, #1 lsrs r6, r3, #1 - orrs r6, r2, r6 + orrs r6, r2 beq 1f movs r6, r0 eors r6, r1 diff --git a/contrib/compiler-rt/lib/builtins/arm/eqdf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/eqdf2vfp.S index 8fa0b2d..d507065 100644 --- a/contrib/compiler-rt/lib/builtins/arm/eqdf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/eqdf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__eqdf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(eq) moveq r0, #1 // set result register to 1 if equal movne r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/eqsf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/eqsf2vfp.S index 3776bf4..fd72b2f 100644 --- a/contrib/compiler-rt/lib/builtins/arm/eqsf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/eqsf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__eqsf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(eq) moveq r0, #1 // set result register to 1 if equal movne r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/gedf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/gedf2vfp.S index 14899f0..364fc5b 100644 --- a/contrib/compiler-rt/lib/builtins/arm/gedf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/gedf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__gedf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(ge) movge r0, #1 // set result register to 1 if greater than or equal movlt r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/gesf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/gesf2vfp.S index b49d04d..346c347 100644 --- a/contrib/compiler-rt/lib/builtins/arm/gesf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/gesf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__gesf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(ge) movge r0, #1 // set result register to 1 if greater than or equal movlt r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/gtdf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/gtdf2vfp.S index 8166305..3389c3a 100644 --- a/contrib/compiler-rt/lib/builtins/arm/gtdf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/gtdf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__gtdf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(gt) movgt r0, #1 // set result register to 1 if equal movle r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/gtsf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/gtsf2vfp.S index d2d8a23..afdba8b 100644 --- a/contrib/compiler-rt/lib/builtins/arm/gtsf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/gtsf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__gtsf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(gt) movgt r0, #1 // set result register to 1 if equal movle r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/ledf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/ledf2vfp.S index a9dab77..4bbe4c8 100644 --- a/contrib/compiler-rt/lib/builtins/arm/ledf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/ledf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__ledf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(ls) movls r0, #1 // set result register to 1 if equal movhi r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/lesf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/lesf2vfp.S index 7e127f4..51232bd 100644 --- a/contrib/compiler-rt/lib/builtins/arm/lesf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/lesf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__lesf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(ls) movls r0, #1 // set result register to 1 if equal movhi r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/ltdf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/ltdf2vfp.S index 8b6f8e4..8e2928c8 100644 --- a/contrib/compiler-rt/lib/builtins/arm/ltdf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/ltdf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__ltdf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(mi) movmi r0, #1 // set result register to 1 if equal movpl r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/ltsf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/ltsf2vfp.S index c4ff812..59c00c6 100644 --- a/contrib/compiler-rt/lib/builtins/arm/ltsf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/ltsf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__ltsf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(mi) movmi r0, #1 // set result register to 1 if equal movpl r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/nedf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/nedf2vfp.S index 7d884e0..aef72eb 100644 --- a/contrib/compiler-rt/lib/builtins/arm/nedf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/nedf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__nedf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(ne) movne r0, #1 // set result register to 0 if unequal moveq r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/nesf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/nesf2vfp.S index 97c764f6..50d60f4 100644 --- a/contrib/compiler-rt/lib/builtins/arm/nesf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/nesf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__nesf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(ne) movne r0, #1 // set result register to 1 if unequal moveq r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/udivsi3.S b/contrib/compiler-rt/lib/builtins/arm/udivsi3.S index 2a02099..82db2cb 100644 --- a/contrib/compiler-rt/lib/builtins/arm/udivsi3.S +++ b/contrib/compiler-rt/lib/builtins/arm/udivsi3.S @@ -36,7 +36,16 @@ DEFINE_COMPILERRT_FUNCTION(__udivsi3) beq LOCAL_LABEL(divby0) udiv r0, r0, r1 bx lr -#else + +LOCAL_LABEL(divby0): + mov r0, #0 +# ifdef __ARM_EABI__ + b __aeabi_idiv0 +# else + JMP(lr) +# endif + +#else /* ! __ARM_ARCH_EXT_IDIV__ */ cmp r1, #1 bcc LOCAL_LABEL(divby0) #if defined(USE_THUMB_1) @@ -185,9 +194,12 @@ LOCAL_LABEL(skip_1): LOCAL_LABEL(divby0): movs r0, #0 # if defined(__ARM_EABI__) + push {r7, lr} bl __aeabi_idiv0 // due to relocation limit, can't use b. -# endif + pop {r7, pc} +# else JMP(lr) +# endif #if defined(USE_THUMB_1) @@ -251,16 +263,6 @@ LOCAL_LABEL(div0block): JMP(lr) #endif /* __ARM_ARCH_EXT_IDIV__ */ -#if __ARM_ARCH_EXT_IDIV__ -LOCAL_LABEL(divby0): - mov r0, #0 -# ifdef __ARM_EABI__ - b __aeabi_idiv0 -# else - JMP(lr) -# endif -#endif - END_COMPILERRT_FUNCTION(__udivsi3) NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/builtins/arm/unorddf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/unorddf2vfp.S index 8556375..6625fa8 100644 --- a/contrib/compiler-rt/lib/builtins/arm/unorddf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/unorddf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__unorddf2vfp) vcmp.f64 d6, d7 #endif vmrs apsr_nzcv, fpscr + ITE(vs) movvs r0, #1 // set result register to 1 if "overflow" (any NaNs) movvc r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/arm/unordsf2vfp.S b/contrib/compiler-rt/lib/builtins/arm/unordsf2vfp.S index 2b16b49..0b5da2b 100644 --- a/contrib/compiler-rt/lib/builtins/arm/unordsf2vfp.S +++ b/contrib/compiler-rt/lib/builtins/arm/unordsf2vfp.S @@ -27,6 +27,7 @@ DEFINE_COMPILERRT_FUNCTION(__unordsf2vfp) vcmp.f32 s14, s15 #endif vmrs apsr_nzcv, fpscr + ITE(vs) movvs r0, #1 // set result register to 1 if "overflow" (any NaNs) movvc r0, #0 bx lr diff --git a/contrib/compiler-rt/lib/builtins/ashldi3.c b/contrib/compiler-rt/lib/builtins/ashldi3.c index eb4698a..fcb0abd 100644 --- a/contrib/compiler-rt/lib/builtins/ashldi3.c +++ b/contrib/compiler-rt/lib/builtins/ashldi3.c @@ -18,8 +18,6 @@ /* Precondition: 0 <= b < bits_in_dword */ -ARM_EABI_FNALIAS(llsl, ashldi3) - COMPILER_RT_ABI di_int __ashldi3(di_int a, si_int b) { @@ -41,3 +39,10 @@ __ashldi3(di_int a, si_int b) } return result.all; } + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int __aeabi_llsl(di_int a, si_int b) { + return __ashldi3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/ashrdi3.c b/contrib/compiler-rt/lib/builtins/ashrdi3.c index 14c878b..b4ab4c6 100644 --- a/contrib/compiler-rt/lib/builtins/ashrdi3.c +++ b/contrib/compiler-rt/lib/builtins/ashrdi3.c @@ -18,8 +18,6 @@ /* Precondition: 0 <= b < bits_in_dword */ -ARM_EABI_FNALIAS(lasr, ashrdi3) - COMPILER_RT_ABI di_int __ashrdi3(di_int a, si_int b) { @@ -42,3 +40,10 @@ __ashrdi3(di_int a, si_int b) } return result.all; } + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int __aeabi_lasr(di_int a, si_int b) { + return __ashrdi3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/assembly.h b/contrib/compiler-rt/lib/builtins/assembly.h index af959b2..4893d483 100644 --- a/contrib/compiler-rt/lib/builtins/assembly.h +++ b/contrib/compiler-rt/lib/builtins/assembly.h @@ -44,7 +44,8 @@ #endif #define CONST_SECTION .section .rodata -#if defined(__GNU__) || defined(__ANDROID__) || defined(__FreeBSD__) +#if defined(__GNU__) || defined(__FreeBSD__) || defined(__Fuchsia__) || \ + defined(__linux__) #define NO_EXEC_STACK_DIRECTIVE .section .note.GNU-stack,"",%progbits #else #define NO_EXEC_STACK_DIRECTIVE @@ -114,10 +115,12 @@ #if defined(USE_THUMB_2) #define IT(cond) it cond #define ITT(cond) itt cond +#define ITE(cond) ite cond #define WIDE(op) op.w #else #define IT(cond) #define ITT(cond) +#define ITE(cond) #define WIDE(op) op #endif #endif /* defined(__arm__) */ diff --git a/contrib/compiler-rt/lib/builtins/bswapdi2.c b/contrib/compiler-rt/lib/builtins/bswapdi2.c new file mode 100644 index 0000000..eb22000 --- /dev/null +++ b/contrib/compiler-rt/lib/builtins/bswapdi2.c @@ -0,0 +1,27 @@ +/* ===-- bswapdi2.c - Implement __bswapdi2 ---------------------------------=== + * + * The LLVM Compiler Infrastructure + * + * This file is dual licensed under the MIT and the University of Illinois Open + * Source Licenses. See LICENSE.TXT for details. + * + * ===----------------------------------------------------------------------=== + * + * This file implements __bswapdi2 for the compiler_rt library. + * + * ===----------------------------------------------------------------------=== + */ + +#include "int_lib.h" + +COMPILER_RT_ABI uint64_t __bswapdi2(uint64_t u) { + return ( + (((u)&0xff00000000000000ULL) >> 56) | + (((u)&0x00ff000000000000ULL) >> 40) | + (((u)&0x0000ff0000000000ULL) >> 24) | + (((u)&0x000000ff00000000ULL) >> 8) | + (((u)&0x00000000ff000000ULL) << 8) | + (((u)&0x0000000000ff0000ULL) << 24) | + (((u)&0x000000000000ff00ULL) << 40) | + (((u)&0x00000000000000ffULL) << 56)); +} diff --git a/contrib/compiler-rt/lib/builtins/bswapsi2.c b/contrib/compiler-rt/lib/builtins/bswapsi2.c new file mode 100644 index 0000000..5d941e6 --- /dev/null +++ b/contrib/compiler-rt/lib/builtins/bswapsi2.c @@ -0,0 +1,23 @@ +/* ===-- bswapsi2.c - Implement __bswapsi2 ---------------------------------=== + * + * The LLVM Compiler Infrastructure + * + * This file is dual licensed under the MIT and the University of Illinois Open + * Source Licenses. See LICENSE.TXT for details. + * + * ===----------------------------------------------------------------------=== + * + * This file implements __bswapsi2 for the compiler_rt library. + * + * ===----------------------------------------------------------------------=== + */ + +#include "int_lib.h" + +COMPILER_RT_ABI uint32_t __bswapsi2(uint32_t u) { + return ( + (((u)&0xff000000) >> 24) | + (((u)&0x00ff0000) >> 8) | + (((u)&0x0000ff00) << 8) | + (((u)&0x000000ff) << 24)); +} diff --git a/contrib/compiler-rt/lib/builtins/clear_cache.c b/contrib/compiler-rt/lib/builtins/clear_cache.c index bb6e247..7ccbe01 100644 --- a/contrib/compiler-rt/lib/builtins/clear_cache.c +++ b/contrib/compiler-rt/lib/builtins/clear_cache.c @@ -82,10 +82,6 @@ uintptr_t GetCurrentProcess(void); #endif #endif -#if defined(__linux__) && defined(__arm__) - #include <asm/unistd.h> -#endif - /* * The compiler generates calls to __clear_cache() when creating * trampoline functions on the stack for use with nested functions. @@ -94,7 +90,7 @@ uintptr_t GetCurrentProcess(void); */ void __clear_cache(void *start, void *end) { -#if __i386__ || __x86_64__ +#if __i386__ || __x86_64__ || defined(_M_IX86) || defined(_M_X64) /* * Intel processors have a unified instruction and data cache * so there is nothing to do @@ -108,6 +104,15 @@ void __clear_cache(void *start, void *end) { sysarch(ARM_SYNC_ICACHE, &arg); #elif defined(__linux__) + /* + * We used to include asm/unistd.h for the __ARM_NR_cacheflush define, but + * it also brought many other unused defines, as well as a dependency on + * kernel headers to be installed. + * + * This value is stable at least since Linux 3.13 and should remain so for + * compatibility reasons, warranting it's re-definition here. + */ + #define __ARM_NR_cacheflush 0x0f0002 register int start_reg __asm("r0") = (int) (intptr_t) start; const register int end_reg __asm("r1") = (int) (intptr_t) end; const register int flags __asm("r2") = 0; diff --git a/contrib/compiler-rt/lib/builtins/comparedf2.c b/contrib/compiler-rt/lib/builtins/comparedf2.c index 9e29752..c5bb169d0 100644 --- a/contrib/compiler-rt/lib/builtins/comparedf2.c +++ b/contrib/compiler-rt/lib/builtins/comparedf2.c @@ -113,8 +113,6 @@ __gedf2(fp_t a, fp_t b) { } } -ARM_EABI_FNALIAS(dcmpun, unorddf2) - COMPILER_RT_ABI int __unorddf2(fp_t a, fp_t b) { const rep_t aAbs = toRep(a) & absMask; @@ -144,3 +142,9 @@ __gtdf2(fp_t a, fp_t b) { return __gedf2(a, b); } +#if defined(__ARM_EABI__) +AEABI_RTABI int __aeabi_dcmpun(fp_t a, fp_t b) { + return __unorddf2(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/comparesf2.c b/contrib/compiler-rt/lib/builtins/comparesf2.c index 1fd5063..4badb5e 100644 --- a/contrib/compiler-rt/lib/builtins/comparesf2.c +++ b/contrib/compiler-rt/lib/builtins/comparesf2.c @@ -113,8 +113,6 @@ __gesf2(fp_t a, fp_t b) { } } -ARM_EABI_FNALIAS(fcmpun, unordsf2) - COMPILER_RT_ABI int __unordsf2(fp_t a, fp_t b) { const rep_t aAbs = toRep(a) & absMask; @@ -143,3 +141,10 @@ COMPILER_RT_ABI enum GE_RESULT __gtsf2(fp_t a, fp_t b) { return __gesf2(a, b); } + +#if defined(__ARM_EABI__) +AEABI_RTABI int __aeabi_fcmpun(fp_t a, fp_t b) { + return __unordsf2(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/cpu_model.c b/contrib/compiler-rt/lib/builtins/cpu_model.c index 9a37370..83ea7a4 100644 --- a/contrib/compiler-rt/lib/builtins/cpu_model.c +++ b/contrib/compiler-rt/lib/builtins/cpu_model.c @@ -27,6 +27,10 @@ #include <intrin.h> #endif +#ifndef __has_attribute +#define __has_attribute(attr) 0 +#endif + enum VendorSignatures { SIG_INTEL = 0x756e6547 /* Genu */, SIG_AMD = 0x68747541 /* Auth */ @@ -40,29 +44,16 @@ enum ProcessorVendors { }; enum ProcessorTypes { - INTEL_ATOM = 1, + INTEL_BONNELL = 1, INTEL_CORE2, INTEL_COREI7, AMDFAM10H, AMDFAM15H, - INTEL_i386, - INTEL_i486, - INTEL_PENTIUM, - INTEL_PENTIUM_PRO, - INTEL_PENTIUM_II, - INTEL_PENTIUM_III, - INTEL_PENTIUM_IV, - INTEL_PENTIUM_M, - INTEL_CORE_DUO, - INTEL_XEONPHI, - INTEL_X86_64, - INTEL_NOCONA, - INTEL_PRESCOTT, - AMD_i486, - AMDPENTIUM, - AMDATHLON, - AMDFAM14H, - AMDFAM16H, + INTEL_SILVERMONT, + INTEL_KNL, + AMD_BTVER1, + AMD_BTVER2, + AMDFAM17H, CPU_TYPE_MAX }; @@ -75,32 +66,14 @@ enum ProcessorSubtypes { AMDFAM10H_ISTANBUL, AMDFAM15H_BDVER1, AMDFAM15H_BDVER2, - INTEL_PENTIUM_MMX, - INTEL_CORE2_65, - INTEL_CORE2_45, + AMDFAM15H_BDVER3, + AMDFAM15H_BDVER4, + AMDFAM17H_ZNVER1, INTEL_COREI7_IVYBRIDGE, INTEL_COREI7_HASWELL, INTEL_COREI7_BROADWELL, INTEL_COREI7_SKYLAKE, INTEL_COREI7_SKYLAKE_AVX512, - INTEL_ATOM_BONNELL, - INTEL_ATOM_SILVERMONT, - INTEL_KNIGHTS_LANDING, - AMDPENTIUM_K6, - AMDPENTIUM_K62, - AMDPENTIUM_K63, - AMDPENTIUM_GEODE, - AMDATHLON_TBIRD, - AMDATHLON_MP, - AMDATHLON_XP, - AMDATHLON_K8SSE3, - AMDATHLON_OPTERON, - AMDATHLON_FX, - AMDATHLON_64, - AMD_BTVER1, - AMD_BTVER2, - AMDFAM15H_BDVER3, - AMDFAM15H_BDVER4, CPU_SUBTYPE_MAX }; @@ -116,11 +89,26 @@ enum ProcessorFeatures { FEATURE_SSE4_2, FEATURE_AVX, FEATURE_AVX2, - FEATURE_AVX512, - FEATURE_AVX512SAVE, - FEATURE_MOVBE, - FEATURE_ADX, - FEATURE_EM64T + FEATURE_SSE4_A, + FEATURE_FMA4, + FEATURE_XOP, + FEATURE_FMA, + FEATURE_AVX512F, + FEATURE_BMI, + FEATURE_BMI2, + FEATURE_AES, + FEATURE_PCLMUL, + FEATURE_AVX512VL, + FEATURE_AVX512BW, + FEATURE_AVX512DQ, + FEATURE_AVX512CD, + FEATURE_AVX512ER, + FEATURE_AVX512PF, + FEATURE_AVX512VBMI, + FEATURE_AVX512IFMA, + FEATURE_AVX5124VNNIW, + FEATURE_AVX5124FMAPS, + FEATURE_AVX512VPOPCNTDQ }; // The check below for i386 was copied from clang's cpuid.h (__get_cpuid_max). @@ -160,26 +148,27 @@ static bool isCpuIdSupported() { /// getX86CpuIDAndInfo - Execute the specified cpuid and return the 4 values in /// the specified arguments. If we can't run cpuid on the host, return true. -static void getX86CpuIDAndInfo(unsigned value, unsigned *rEAX, unsigned *rEBX, +static bool getX86CpuIDAndInfo(unsigned value, unsigned *rEAX, unsigned *rEBX, unsigned *rECX, unsigned *rEDX) { #if defined(__GNUC__) || defined(__clang__) #if defined(__x86_64__) - // gcc doesn't know cpuid would clobber ebx/rbx. Preseve it manually. + // gcc doesn't know cpuid would clobber ebx/rbx. Preserve it manually. + // FIXME: should we save this for Clang? __asm__("movq\t%%rbx, %%rsi\n\t" "cpuid\n\t" "xchgq\t%%rbx, %%rsi\n\t" : "=a"(*rEAX), "=S"(*rEBX), "=c"(*rECX), "=d"(*rEDX) : "a"(value)); + return false; #elif defined(__i386__) __asm__("movl\t%%ebx, %%esi\n\t" "cpuid\n\t" "xchgl\t%%ebx, %%esi\n\t" : "=a"(*rEAX), "=S"(*rEBX), "=c"(*rECX), "=d"(*rEDX) : "a"(value)); -// pedantic #else returns to appease -Wunreachable-code (so we don't generate -// postprocessed code that looks like "return true; return false;") + return false; #else - assert(0 && "This method is defined only for x86."); + return true; #endif #elif defined(_MSC_VER) // The MSVC intrinsic is portable across x86 and x64. @@ -189,19 +178,20 @@ static void getX86CpuIDAndInfo(unsigned value, unsigned *rEAX, unsigned *rEBX, *rEBX = registers[1]; *rECX = registers[2]; *rEDX = registers[3]; + return false; #else - assert(0 && "This method is defined only for GNUC, Clang or MSVC."); + return true; #endif } /// getX86CpuIDAndInfoEx - Execute the specified cpuid with subleaf and return /// the 4 values in the specified arguments. If we can't run cpuid on the host, /// return true. -static void getX86CpuIDAndInfoEx(unsigned value, unsigned subleaf, +static bool getX86CpuIDAndInfoEx(unsigned value, unsigned subleaf, unsigned *rEAX, unsigned *rEBX, unsigned *rECX, unsigned *rEDX) { -#if defined(__x86_64__) || defined(_M_X64) #if defined(__GNUC__) || defined(__clang__) +#if defined(__x86_64__) // gcc doesn't know cpuid would clobber ebx/rbx. Preserve it manually. // FIXME: should we save this for Clang? __asm__("movq\t%%rbx, %%rsi\n\t" @@ -209,42 +199,27 @@ static void getX86CpuIDAndInfoEx(unsigned value, unsigned subleaf, "xchgq\t%%rbx, %%rsi\n\t" : "=a"(*rEAX), "=S"(*rEBX), "=c"(*rECX), "=d"(*rEDX) : "a"(value), "c"(subleaf)); -#elif defined(_MSC_VER) - int registers[4]; - __cpuidex(registers, value, subleaf); - *rEAX = registers[0]; - *rEBX = registers[1]; - *rECX = registers[2]; - *rEDX = registers[3]; -#else - assert(0 && "This method is defined only for GNUC, Clang or MSVC."); -#endif -#elif defined(__i386__) || defined(_M_IX86) -#if defined(__GNUC__) || defined(__clang__) + return false; +#elif defined(__i386__) __asm__("movl\t%%ebx, %%esi\n\t" "cpuid\n\t" "xchgl\t%%ebx, %%esi\n\t" : "=a"(*rEAX), "=S"(*rEBX), "=c"(*rECX), "=d"(*rEDX) : "a"(value), "c"(subleaf)); -#elif defined(_MSC_VER) - __asm { - mov eax,value - mov ecx,subleaf - cpuid - mov esi,rEAX - mov dword ptr [esi],eax - mov esi,rEBX - mov dword ptr [esi],ebx - mov esi,rECX - mov dword ptr [esi],ecx - mov esi,rEDX - mov dword ptr [esi],edx - } + return false; #else - assert(0 && "This method is defined only for GNUC, Clang or MSVC."); + return true; #endif +#elif defined(_MSC_VER) + int registers[4]; + __cpuidex(registers, value, subleaf); + *rEAX = registers[0]; + *rEBX = registers[1]; + *rECX = registers[2]; + *rEDX = registers[3]; + return false; #else - assert(0 && "This method is defined only for x86."); + return true; #endif } @@ -279,84 +254,15 @@ static void detectX86FamilyModel(unsigned EAX, unsigned *Family, } } -static void getIntelProcessorTypeAndSubtype(unsigned int Family, - unsigned int Model, - unsigned int Brand_id, - unsigned int Features, - unsigned *Type, unsigned *Subtype) { +static void +getIntelProcessorTypeAndSubtype(unsigned Family, unsigned Model, + unsigned Brand_id, unsigned Features, + unsigned *Type, unsigned *Subtype) { if (Brand_id != 0) return; switch (Family) { - case 3: - *Type = INTEL_i386; - break; - case 4: - switch (Model) { - case 0: // Intel486 DX processors - case 1: // Intel486 DX processors - case 2: // Intel486 SX processors - case 3: // Intel487 processors, IntelDX2 OverDrive processors, - // IntelDX2 processors - case 4: // Intel486 SL processor - case 5: // IntelSX2 processors - case 7: // Write-Back Enhanced IntelDX2 processors - case 8: // IntelDX4 OverDrive processors, IntelDX4 processors - default: - *Type = INTEL_i486; - break; - } - case 5: - switch (Model) { - case 1: // Pentium OverDrive processor for Pentium processor (60, 66), - // Pentium processors (60, 66) - case 2: // Pentium OverDrive processor for Pentium processor (75, 90, - // 100, 120, 133), Pentium processors (75, 90, 100, 120, 133, - // 150, 166, 200) - case 3: // Pentium OverDrive processors for Intel486 processor-based - // systems - *Type = INTEL_PENTIUM; - break; - case 4: // Pentium OverDrive processor with MMX technology for Pentium - // processor (75, 90, 100, 120, 133), Pentium processor with - // MMX technology (166, 200) - *Type = INTEL_PENTIUM; - *Subtype = INTEL_PENTIUM_MMX; - break; - default: - *Type = INTEL_PENTIUM; - break; - } case 6: switch (Model) { - case 0x01: // Pentium Pro processor - *Type = INTEL_PENTIUM_PRO; - break; - case 0x03: // Intel Pentium II OverDrive processor, Pentium II processor, - // model 03 - case 0x05: // Pentium II processor, model 05, Pentium II Xeon processor, - // model 05, and Intel Celeron processor, model 05 - case 0x06: // Celeron processor, model 06 - *Type = INTEL_PENTIUM_II; - break; - case 0x07: // Pentium III processor, model 07, and Pentium III Xeon - // processor, model 07 - case 0x08: // Pentium III processor, model 08, Pentium III Xeon processor, - // model 08, and Celeron processor, model 08 - case 0x0a: // Pentium III Xeon processor, model 0Ah - case 0x0b: // Pentium III processor, model 0Bh - *Type = INTEL_PENTIUM_III; - break; - case 0x09: // Intel Pentium M processor, Intel Celeron M processor model 09. - case 0x0d: // Intel Pentium M processor, Intel Celeron M processor, model - // 0Dh. All processors are manufactured using the 90 nm process. - case 0x15: // Intel EP80579 Integrated Processor and Intel EP80579 - // Integrated Processor with Intel QuickAssist Technology - *Type = INTEL_PENTIUM_M; - break; - case 0x0e: // Intel Core Duo processor, Intel Core Solo processor, model - // 0Eh. All processors are manufactured using the 65 nm process. - *Type = INTEL_CORE_DUO; - break; // yonah case 0x0f: // Intel Core 2 Duo processor, Intel Core 2 Duo mobile // processor, Intel Core 2 Quad processor, Intel Core 2 Quad // mobile processor, Intel Core 2 Extreme processor, Intel @@ -364,9 +270,6 @@ static void getIntelProcessorTypeAndSubtype(unsigned int Family, // 0Fh. All processors are manufactured using the 65 nm process. case 0x16: // Intel Celeron processor model 16h. All processors are // manufactured using the 65 nm process - *Type = INTEL_CORE2; // "core2" - *Subtype = INTEL_CORE2_65; - break; case 0x17: // Intel Core 2 Extreme processor, Intel Xeon processor, model // 17h. All processors are manufactured using the 45 nm process. // @@ -374,14 +277,13 @@ static void getIntelProcessorTypeAndSubtype(unsigned int Family, case 0x1d: // Intel Xeon processor MP. All processors are manufactured using // the 45 nm process. *Type = INTEL_CORE2; // "penryn" - *Subtype = INTEL_CORE2_45; break; case 0x1a: // Intel Core i7 processor and Intel Xeon processor. All // processors are manufactured using the 45 nm process. case 0x1e: // Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz. // As found in a Summer 2010 model iMac. case 0x1f: - case 0x2e: // Nehalem EX + case 0x2e: // Nehalem EX *Type = INTEL_COREI7; // "nehalem" *Subtype = INTEL_COREI7_NEHALEM; break; @@ -399,7 +301,7 @@ static void getIntelProcessorTypeAndSubtype(unsigned int Family, *Subtype = INTEL_COREI7_SANDYBRIDGE; break; case 0x3a: - case 0x3e: // Ivy Bridge EP + case 0x3e: // Ivy Bridge EP *Type = INTEL_COREI7; // "ivybridge" *Subtype = INTEL_COREI7_IVYBRIDGE; break; @@ -423,22 +325,26 @@ static void getIntelProcessorTypeAndSubtype(unsigned int Family, break; // Skylake: - case 0x4e: - *Type = INTEL_COREI7; // "skylake-avx512" - *Subtype = INTEL_COREI7_SKYLAKE_AVX512; - break; - case 0x5e: + case 0x4e: // Skylake mobile + case 0x5e: // Skylake desktop + case 0x8e: // Kaby Lake mobile + case 0x9e: // Kaby Lake desktop *Type = INTEL_COREI7; // "skylake" *Subtype = INTEL_COREI7_SKYLAKE; break; + // Skylake Xeon: + case 0x55: + *Type = INTEL_COREI7; + *Subtype = INTEL_COREI7_SKYLAKE_AVX512; // "skylake-avx512" + break; + case 0x1c: // Most 45 nm Intel Atom processors case 0x26: // 45 nm Atom Lincroft case 0x27: // 32 nm Atom Medfield case 0x35: // 32 nm Atom Midview case 0x36: // 32 nm Atom Midview - *Type = INTEL_ATOM; - *Subtype = INTEL_ATOM_BONNELL; + *Type = INTEL_BONNELL; break; // "bonnell" // Atom Silvermont codes from the Intel software optimization guide. @@ -448,185 +354,29 @@ static void getIntelProcessorTypeAndSubtype(unsigned int Family, case 0x5a: case 0x5d: case 0x4c: // really airmont - *Type = INTEL_ATOM; - *Subtype = INTEL_ATOM_SILVERMONT; + *Type = INTEL_SILVERMONT; break; // "silvermont" case 0x57: - *Type = INTEL_XEONPHI; // knl - *Subtype = INTEL_KNIGHTS_LANDING; + *Type = INTEL_KNL; // knl break; - default: // Unknown family 6 CPU, try to guess. - if (Features & (1 << FEATURE_AVX512)) { - *Type = INTEL_XEONPHI; // knl - *Subtype = INTEL_KNIGHTS_LANDING; - break; - } - if (Features & (1 << FEATURE_ADX)) { - *Type = INTEL_COREI7; - *Subtype = INTEL_COREI7_BROADWELL; - break; - } - if (Features & (1 << FEATURE_AVX2)) { - *Type = INTEL_COREI7; - *Subtype = INTEL_COREI7_HASWELL; - break; - } - if (Features & (1 << FEATURE_AVX)) { - *Type = INTEL_COREI7; - *Subtype = INTEL_COREI7_SANDYBRIDGE; - break; - } - if (Features & (1 << FEATURE_SSE4_2)) { - if (Features & (1 << FEATURE_MOVBE)) { - *Type = INTEL_ATOM; - *Subtype = INTEL_ATOM_SILVERMONT; - } else { - *Type = INTEL_COREI7; - *Subtype = INTEL_COREI7_NEHALEM; - } - break; - } - if (Features & (1 << FEATURE_SSE4_1)) { - *Type = INTEL_CORE2; // "penryn" - *Subtype = INTEL_CORE2_45; - break; - } - if (Features & (1 << FEATURE_SSSE3)) { - if (Features & (1 << FEATURE_MOVBE)) { - *Type = INTEL_ATOM; - *Subtype = INTEL_ATOM_BONNELL; // "bonnell" - } else { - *Type = INTEL_CORE2; // "core2" - *Subtype = INTEL_CORE2_65; - } - break; - } - if (Features & (1 << FEATURE_EM64T)) { - *Type = INTEL_X86_64; - break; // x86-64 - } - if (Features & (1 << FEATURE_SSE2)) { - *Type = INTEL_PENTIUM_M; - break; - } - if (Features & (1 << FEATURE_SSE)) { - *Type = INTEL_PENTIUM_III; - break; - } - if (Features & (1 << FEATURE_MMX)) { - *Type = INTEL_PENTIUM_II; - break; - } - *Type = INTEL_PENTIUM_PRO; - break; - } - case 15: { - switch (Model) { - case 0: // Pentium 4 processor, Intel Xeon processor. All processors are - // model 00h and manufactured using the 0.18 micron process. - case 1: // Pentium 4 processor, Intel Xeon processor, Intel Xeon - // processor MP, and Intel Celeron processor. All processors are - // model 01h and manufactured using the 0.18 micron process. - case 2: // Pentium 4 processor, Mobile Intel Pentium 4 processor - M, - // Intel Xeon processor, Intel Xeon processor MP, Intel Celeron - // processor, and Mobile Intel Celeron processor. All processors - // are model 02h and manufactured using the 0.13 micron process. - *Type = - ((Features & (1 << FEATURE_EM64T)) ? INTEL_X86_64 : INTEL_PENTIUM_IV); - break; - - case 3: // Pentium 4 processor, Intel Xeon processor, Intel Celeron D - // processor. All processors are model 03h and manufactured using - // the 90 nm process. - case 4: // Pentium 4 processor, Pentium 4 processor Extreme Edition, - // Pentium D processor, Intel Xeon processor, Intel Xeon - // processor MP, Intel Celeron D processor. All processors are - // model 04h and manufactured using the 90 nm process. - case 6: // Pentium 4 processor, Pentium D processor, Pentium processor - // Extreme Edition, Intel Xeon processor, Intel Xeon processor - // MP, Intel Celeron D processor. All processors are model 06h - // and manufactured using the 65 nm process. - *Type = - ((Features & (1 << FEATURE_EM64T)) ? INTEL_NOCONA : INTEL_PRESCOTT); - break; - - default: - *Type = - ((Features & (1 << FEATURE_EM64T)) ? INTEL_X86_64 : INTEL_PENTIUM_IV); + default: // Unknown family 6 CPU. break; + break; } - } default: - break; /*"generic"*/ + break; // Unknown. } } -static void getAMDProcessorTypeAndSubtype(unsigned int Family, - unsigned int Model, - unsigned int Features, unsigned *Type, +static void getAMDProcessorTypeAndSubtype(unsigned Family, unsigned Model, + unsigned Features, unsigned *Type, unsigned *Subtype) { // FIXME: this poorly matches the generated SubtargetFeatureKV table. There // appears to be no way to generate the wide variety of AMD-specific targets // from the information returned from CPUID. switch (Family) { - case 4: - *Type = AMD_i486; - case 5: - *Type = AMDPENTIUM; - switch (Model) { - case 6: - case 7: - *Subtype = AMDPENTIUM_K6; - break; // "k6" - case 8: - *Subtype = AMDPENTIUM_K62; - break; // "k6-2" - case 9: - case 13: - *Subtype = AMDPENTIUM_K63; - break; // "k6-3" - case 10: - *Subtype = AMDPENTIUM_GEODE; - break; // "geode" - default: - break; - } - case 6: - *Type = AMDATHLON; - switch (Model) { - case 4: - *Subtype = AMDATHLON_TBIRD; - break; // "athlon-tbird" - case 6: - case 7: - case 8: - *Subtype = AMDATHLON_MP; - break; // "athlon-mp" - case 10: - *Subtype = AMDATHLON_XP; - break; // "athlon-xp" - default: - break; - } - case 15: - *Type = AMDATHLON; - if (Features & (1 << FEATURE_SSE3)) { - *Subtype = AMDATHLON_K8SSE3; - break; // "k8-sse3" - } - switch (Model) { - case 1: - *Subtype = AMDATHLON_OPTERON; - break; // "opteron" - case 5: - *Subtype = AMDATHLON_FX; - break; // "athlon-fx"; also opteron - default: - *Subtype = AMDATHLON_64; - break; // "athlon64" - } case 16: *Type = AMDFAM10H; // "amdfam10" switch (Model) { @@ -639,23 +389,16 @@ static void getAMDProcessorTypeAndSubtype(unsigned int Family, case 8: *Subtype = AMDFAM10H_ISTANBUL; break; - default: - break; } + break; case 20: - *Type = AMDFAM14H; - *Subtype = AMD_BTVER1; + *Type = AMD_BTVER1; break; // "btver1"; case 21: *Type = AMDFAM15H; - if (!(Features & - (1 << FEATURE_AVX))) { // If no AVX support, provide a sane fallback. - *Subtype = AMD_BTVER1; - break; // "btver1" - } - if (Model >= 0x50 && Model <= 0x6f) { + if (Model >= 0x60 && Model <= 0x7f) { *Subtype = AMDFAM15H_BDVER4; - break; // "bdver4"; 50h-6Fh: Excavator + break; // "bdver4"; 60h-7Fh: Excavator } if (Model >= 0x30 && Model <= 0x3f) { *Subtype = AMDFAM15H_BDVER3; @@ -671,31 +414,47 @@ static void getAMDProcessorTypeAndSubtype(unsigned int Family, } break; case 22: - *Type = AMDFAM16H; - if (!(Features & - (1 << FEATURE_AVX))) { // If no AVX support provide a sane fallback. - *Subtype = AMD_BTVER1; - break; // "btver1"; - } - *Subtype = AMD_BTVER2; + *Type = AMD_BTVER2; break; // "btver2" + case 23: + *Type = AMDFAM17H; + *Subtype = AMDFAM17H_ZNVER1; + break; default: break; // "generic" } } -static unsigned getAvailableFeatures(unsigned int ECX, unsigned int EDX, - unsigned MaxLeaf) { +static void getAvailableFeatures(unsigned ECX, unsigned EDX, unsigned MaxLeaf, + unsigned *FeaturesOut) { unsigned Features = 0; - unsigned int EAX, EBX; - Features |= (((EDX >> 23) & 1) << FEATURE_MMX); - Features |= (((EDX >> 25) & 1) << FEATURE_SSE); - Features |= (((EDX >> 26) & 1) << FEATURE_SSE2); - Features |= (((ECX >> 0) & 1) << FEATURE_SSE3); - Features |= (((ECX >> 9) & 1) << FEATURE_SSSE3); - Features |= (((ECX >> 19) & 1) << FEATURE_SSE4_1); - Features |= (((ECX >> 20) & 1) << FEATURE_SSE4_2); - Features |= (((ECX >> 22) & 1) << FEATURE_MOVBE); + unsigned EAX, EBX; + + if ((EDX >> 15) & 1) + Features |= 1 << FEATURE_CMOV; + if ((EDX >> 23) & 1) + Features |= 1 << FEATURE_MMX; + if ((EDX >> 25) & 1) + Features |= 1 << FEATURE_SSE; + if ((EDX >> 26) & 1) + Features |= 1 << FEATURE_SSE2; + + if ((ECX >> 0) & 1) + Features |= 1 << FEATURE_SSE3; + if ((ECX >> 1) & 1) + Features |= 1 << FEATURE_PCLMUL; + if ((ECX >> 9) & 1) + Features |= 1 << FEATURE_SSSE3; + if ((ECX >> 12) & 1) + Features |= 1 << FEATURE_FMA; + if ((ECX >> 19) & 1) + Features |= 1 << FEATURE_SSE4_1; + if ((ECX >> 20) & 1) + Features |= 1 << FEATURE_SSE4_2; + if ((ECX >> 23) & 1) + Features |= 1 << FEATURE_POPCNT; + if ((ECX >> 25) & 1) + Features |= 1 << FEATURE_AES; // If CPUID indicates support for XSAVE, XRESTORE and AVX, and XGETBV // indicates that the AVX registers will be saved and restored on context @@ -704,30 +463,72 @@ static unsigned getAvailableFeatures(unsigned int ECX, unsigned int EDX, bool HasAVX = ((ECX & AVXBits) == AVXBits) && !getX86XCR0(&EAX, &EDX) && ((EAX & 0x6) == 0x6); bool HasAVX512Save = HasAVX && ((EAX & 0xe0) == 0xe0); - bool HasLeaf7 = MaxLeaf >= 0x7; - getX86CpuIDAndInfoEx(0x7, 0x0, &EAX, &EBX, &ECX, &EDX); - bool HasADX = HasLeaf7 && ((EBX >> 19) & 1); - bool HasAVX2 = HasAVX && HasLeaf7 && (EBX & 0x20); - bool HasAVX512 = HasLeaf7 && HasAVX512Save && ((EBX >> 16) & 1); - Features |= (HasAVX << FEATURE_AVX); - Features |= (HasAVX2 << FEATURE_AVX2); - Features |= (HasAVX512 << FEATURE_AVX512); - Features |= (HasAVX512Save << FEATURE_AVX512SAVE); - Features |= (HasADX << FEATURE_ADX); - - getX86CpuIDAndInfo(0x80000001, &EAX, &EBX, &ECX, &EDX); - Features |= (((EDX >> 29) & 0x1) << FEATURE_EM64T); - return Features; + + if (HasAVX) + Features |= 1 << FEATURE_AVX; + + bool HasLeaf7 = + MaxLeaf >= 0x7 && !getX86CpuIDAndInfoEx(0x7, 0x0, &EAX, &EBX, &ECX, &EDX); + + if (HasLeaf7 && ((EBX >> 3) & 1)) + Features |= 1 << FEATURE_BMI; + if (HasLeaf7 && ((EBX >> 5) & 1) && HasAVX) + Features |= 1 << FEATURE_AVX2; + if (HasLeaf7 && ((EBX >> 9) & 1)) + Features |= 1 << FEATURE_BMI2; + if (HasLeaf7 && ((EBX >> 16) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512F; + if (HasLeaf7 && ((EBX >> 17) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512DQ; + if (HasLeaf7 && ((EBX >> 21) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512IFMA; + if (HasLeaf7 && ((EBX >> 26) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512PF; + if (HasLeaf7 && ((EBX >> 27) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512ER; + if (HasLeaf7 && ((EBX >> 28) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512CD; + if (HasLeaf7 && ((EBX >> 30) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512BW; + if (HasLeaf7 && ((EBX >> 31) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512VL; + + if (HasLeaf7 && ((ECX >> 1) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512VBMI; + if (HasLeaf7 && ((ECX >> 14) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX512VPOPCNTDQ; + + if (HasLeaf7 && ((EDX >> 2) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX5124VNNIW; + if (HasLeaf7 && ((EDX >> 3) & 1) && HasAVX512Save) + Features |= 1 << FEATURE_AVX5124FMAPS; + + unsigned MaxExtLevel; + getX86CpuIDAndInfo(0x80000000, &MaxExtLevel, &EBX, &ECX, &EDX); + + bool HasExtLeaf1 = MaxExtLevel >= 0x80000001 && + !getX86CpuIDAndInfo(0x80000001, &EAX, &EBX, &ECX, &EDX); + if (HasExtLeaf1 && ((ECX >> 6) & 1)) + Features |= 1 << FEATURE_SSE4_A; + if (HasExtLeaf1 && ((ECX >> 11) & 1)) + Features |= 1 << FEATURE_XOP; + if (HasExtLeaf1 && ((ECX >> 16) & 1)) + Features |= 1 << FEATURE_FMA4; + + *FeaturesOut = Features; } -#ifdef HAVE_INIT_PRIORITY -#define CONSTRUCTOR_PRIORITY (101) +#if defined(HAVE_INIT_PRIORITY) +#define CONSTRUCTOR_ATTRIBUTE __attribute__((__constructor__ 101)) +#elif __has_attribute(__constructor__) +#define CONSTRUCTOR_ATTRIBUTE __attribute__((__constructor__)) #else -#define CONSTRUCTOR_PRIORITY +// FIXME: For MSVC, we should make a function pointer global in .CRT$X?? so that +// this runs during initialization. +#define CONSTRUCTOR_ATTRIBUTE #endif -int __cpu_indicator_init(void) - __attribute__((constructor CONSTRUCTOR_PRIORITY)); +int __cpu_indicator_init(void) CONSTRUCTOR_ATTRIBUTE; struct __processor_model { unsigned int __cpu_vendor; @@ -742,13 +543,13 @@ struct __processor_model { the priority set. However, it still runs after ifunc initializers and needs to be called explicitly there. */ -int __attribute__((constructor CONSTRUCTOR_PRIORITY)) +int CONSTRUCTOR_ATTRIBUTE __cpu_indicator_init(void) { - unsigned int EAX, EBX, ECX, EDX; - unsigned int MaxLeaf = 5; - unsigned int Vendor; - unsigned int Model, Family, Brand_id; - unsigned int Features = 0; + unsigned EAX, EBX, ECX, EDX; + unsigned MaxLeaf = 5; + unsigned Vendor; + unsigned Model, Family, Brand_id; + unsigned Features = 0; /* This function needs to run just once. */ if (__cpu_model.__cpu_vendor) @@ -758,9 +559,7 @@ __cpu_indicator_init(void) { return -1; /* Assume cpuid insn present. Run in level 0 to get vendor id. */ - getX86CpuIDAndInfo(0, &MaxLeaf, &Vendor, &ECX, &EDX); - - if (MaxLeaf < 1) { + if (getX86CpuIDAndInfo(0, &MaxLeaf, &Vendor, &ECX, &EDX) || MaxLeaf < 1) { __cpu_model.__cpu_vendor = VENDOR_OTHER; return -1; } @@ -769,7 +568,7 @@ __cpu_indicator_init(void) { Brand_id = EBX & 0xff; /* Find available features. */ - Features = getAvailableFeatures(ECX, EDX, MaxLeaf); + getAvailableFeatures(ECX, EDX, MaxLeaf, &Features); __cpu_model.__cpu_features[0] = Features; if (Vendor == SIG_INTEL) { diff --git a/contrib/compiler-rt/lib/builtins/divdf3.c b/contrib/compiler-rt/lib/builtins/divdf3.c index ab44c2b..492e32b 100644 --- a/contrib/compiler-rt/lib/builtins/divdf3.c +++ b/contrib/compiler-rt/lib/builtins/divdf3.c @@ -19,8 +19,6 @@ #define DOUBLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(ddiv, divdf3) - COMPILER_RT_ABI fp_t __divdf3(fp_t a, fp_t b) { @@ -183,3 +181,10 @@ __divdf3(fp_t a, fp_t b) { return result; } } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_ddiv(fp_t a, fp_t b) { + return __divdf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/divsf3.c b/contrib/compiler-rt/lib/builtins/divsf3.c index de2e376..aa6289a 100644 --- a/contrib/compiler-rt/lib/builtins/divsf3.c +++ b/contrib/compiler-rt/lib/builtins/divsf3.c @@ -19,8 +19,6 @@ #define SINGLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(fdiv, divsf3) - COMPILER_RT_ABI fp_t __divsf3(fp_t a, fp_t b) { @@ -167,3 +165,10 @@ __divsf3(fp_t a, fp_t b) { return fromRep(absResult | quotientSign); } } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_fdiv(fp_t a, fp_t b) { + return __divsf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/divsi3.c b/contrib/compiler-rt/lib/builtins/divsi3.c index bab4aef..3852e39 100644 --- a/contrib/compiler-rt/lib/builtins/divsi3.c +++ b/contrib/compiler-rt/lib/builtins/divsi3.c @@ -16,8 +16,6 @@ /* Returns: a / b */ -ARM_EABI_FNALIAS(idiv, divsi3) - COMPILER_RT_ABI si_int __divsi3(si_int a, si_int b) { @@ -35,3 +33,10 @@ __divsi3(si_int a, si_int b) */ return ((su_int)a/(su_int)b ^ s_a) - s_a; /* negate if s_a == -1 */ } + +#if defined(__ARM_EABI__) +AEABI_RTABI si_int __aeabi_idiv(si_int a, si_int b) { + return __divsi3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/divtc3.c b/contrib/compiler-rt/lib/builtins/divtc3.c index 04693df..16e538b 100644 --- a/contrib/compiler-rt/lib/builtins/divtc3.c +++ b/contrib/compiler-rt/lib/builtins/divtc3.c @@ -17,7 +17,7 @@ /* Returns: the quotient of (a + ib) / (c + id) */ -COMPILER_RT_ABI long double _Complex +COMPILER_RT_ABI Lcomplex __divtc3(long double __a, long double __b, long double __c, long double __d) { int __ilogbw = 0; @@ -29,31 +29,31 @@ __divtc3(long double __a, long double __b, long double __c, long double __d) __d = crt_scalbnl(__d, -__ilogbw); } long double __denom = __c * __c + __d * __d; - long double _Complex z; - __real__ z = crt_scalbnl((__a * __c + __b * __d) / __denom, -__ilogbw); - __imag__ z = crt_scalbnl((__b * __c - __a * __d) / __denom, -__ilogbw); - if (crt_isnan(__real__ z) && crt_isnan(__imag__ z)) + Lcomplex z; + COMPLEX_REAL(z) = crt_scalbnl((__a * __c + __b * __d) / __denom, -__ilogbw); + COMPLEX_IMAGINARY(z) = crt_scalbnl((__b * __c - __a * __d) / __denom, -__ilogbw); + if (crt_isnan(COMPLEX_REAL(z)) && crt_isnan(COMPLEX_IMAGINARY(z))) { if ((__denom == 0.0) && (!crt_isnan(__a) || !crt_isnan(__b))) { - __real__ z = crt_copysignl(CRT_INFINITY, __c) * __a; - __imag__ z = crt_copysignl(CRT_INFINITY, __c) * __b; + COMPLEX_REAL(z) = crt_copysignl(CRT_INFINITY, __c) * __a; + COMPLEX_IMAGINARY(z) = crt_copysignl(CRT_INFINITY, __c) * __b; } else if ((crt_isinf(__a) || crt_isinf(__b)) && crt_isfinite(__c) && crt_isfinite(__d)) { __a = crt_copysignl(crt_isinf(__a) ? 1.0 : 0.0, __a); __b = crt_copysignl(crt_isinf(__b) ? 1.0 : 0.0, __b); - __real__ z = CRT_INFINITY * (__a * __c + __b * __d); - __imag__ z = CRT_INFINITY * (__b * __c - __a * __d); + COMPLEX_REAL(z) = CRT_INFINITY * (__a * __c + __b * __d); + COMPLEX_IMAGINARY(z) = CRT_INFINITY * (__b * __c - __a * __d); } else if (crt_isinf(__logbw) && __logbw > 0.0 && crt_isfinite(__a) && crt_isfinite(__b)) { __c = crt_copysignl(crt_isinf(__c) ? 1.0 : 0.0, __c); __d = crt_copysignl(crt_isinf(__d) ? 1.0 : 0.0, __d); - __real__ z = 0.0 * (__a * __c + __b * __d); - __imag__ z = 0.0 * (__b * __c - __a * __d); + COMPLEX_REAL(z) = 0.0 * (__a * __c + __b * __d); + COMPLEX_IMAGINARY(z) = 0.0 * (__b * __c - __a * __d); } } return z; diff --git a/contrib/compiler-rt/lib/builtins/emutls.c b/contrib/compiler-rt/lib/builtins/emutls.c index eccbf53..12aad3a 100644 --- a/contrib/compiler-rt/lib/builtins/emutls.c +++ b/contrib/compiler-rt/lib/builtins/emutls.c @@ -7,7 +7,6 @@ * * ===----------------------------------------------------------------------=== */ -#include <pthread.h> #include <stdint.h> #include <stdlib.h> #include <string.h> @@ -15,6 +14,23 @@ #include "int_lib.h" #include "int_util.h" +typedef struct emutls_address_array { + uintptr_t size; /* number of elements in the 'data' array */ + void* data[]; +} emutls_address_array; + +static void emutls_shutdown(emutls_address_array *array); + +#ifndef _WIN32 + +#include <pthread.h> + +static pthread_mutex_t emutls_mutex = PTHREAD_MUTEX_INITIALIZER; +static pthread_key_t emutls_pthread_key; + +typedef unsigned int gcc_word __attribute__((mode(word))); +typedef unsigned int gcc_pointer __attribute__((mode(pointer))); + /* Default is not to use posix_memalign, so systems like Android * can use thread local data without heavier POSIX memory allocators. */ @@ -22,26 +38,6 @@ #define EMUTLS_USE_POSIX_MEMALIGN 0 #endif -/* For every TLS variable xyz, - * there is one __emutls_control variable named __emutls_v.xyz. - * If xyz has non-zero initial value, __emutls_v.xyz's "value" - * will point to __emutls_t.xyz, which has the initial value. - */ -typedef unsigned int gcc_word __attribute__((mode(word))); -typedef struct __emutls_control { - /* Must use gcc_word here, instead of size_t, to match GCC. When - gcc_word is larger than size_t, the upper extra bits are all - zeros. We can use variables of size_t to operate on size and - align. */ - gcc_word size; /* size of the object in bytes */ - gcc_word align; /* alignment of the object in bytes */ - union { - uintptr_t index; /* data[index-1] is the object address */ - void* address; /* object address, when in single thread env */ - } object; - void* value; /* null or non-zero initial value for the object */ -} __emutls_control; - static __inline void *emutls_memalign_alloc(size_t align, size_t size) { void *base; #if EMUTLS_USE_POSIX_MEMALIGN @@ -50,7 +46,7 @@ static __inline void *emutls_memalign_alloc(size_t align, size_t size) { #else #define EXTRA_ALIGN_PTR_BYTES (align - 1 + sizeof(void*)) char* object; - if ((object = malloc(EXTRA_ALIGN_PTR_BYTES + size)) == NULL) + if ((object = (char*)malloc(EXTRA_ALIGN_PTR_BYTES + size)) == NULL) abort(); base = (void*)(((uintptr_t)(object + EXTRA_ALIGN_PTR_BYTES)) & ~(uintptr_t)(align - 1)); @@ -69,10 +65,207 @@ static __inline void emutls_memalign_free(void *base) { #endif } +static void emutls_key_destructor(void* ptr) { + emutls_shutdown((emutls_address_array*)ptr); + free(ptr); +} + +static __inline void emutls_init(void) { + if (pthread_key_create(&emutls_pthread_key, emutls_key_destructor) != 0) + abort(); +} + +static __inline void emutls_init_once(void) { + static pthread_once_t once = PTHREAD_ONCE_INIT; + pthread_once(&once, emutls_init); +} + +static __inline void emutls_lock() { + pthread_mutex_lock(&emutls_mutex); +} + +static __inline void emutls_unlock() { + pthread_mutex_unlock(&emutls_mutex); +} + +static __inline void emutls_setspecific(emutls_address_array *value) { + pthread_setspecific(emutls_pthread_key, (void*) value); +} + +static __inline emutls_address_array* emutls_getspecific() { + return (emutls_address_array*) pthread_getspecific(emutls_pthread_key); +} + +#else + +#include <windows.h> +#include <malloc.h> +#include <stdio.h> +#include <assert.h> +#include <immintrin.h> + +static LPCRITICAL_SECTION emutls_mutex; +static DWORD emutls_tls_index = TLS_OUT_OF_INDEXES; + +typedef uintptr_t gcc_word; +typedef void * gcc_pointer; + +static void win_error(DWORD last_err, const char *hint) { + char *buffer = NULL; + if (FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | + FORMAT_MESSAGE_FROM_SYSTEM | + FORMAT_MESSAGE_MAX_WIDTH_MASK, + NULL, last_err, 0, (LPSTR)&buffer, 1, NULL)) { + fprintf(stderr, "Windows error: %s\n", buffer); + } else { + fprintf(stderr, "Unkown Windows error: %s\n", hint); + } + LocalFree(buffer); +} + +static __inline void win_abort(DWORD last_err, const char *hint) { + win_error(last_err, hint); + abort(); +} + +static __inline void *emutls_memalign_alloc(size_t align, size_t size) { + void *base = _aligned_malloc(size, align); + if (!base) + win_abort(GetLastError(), "_aligned_malloc"); + return base; +} + +static __inline void emutls_memalign_free(void *base) { + _aligned_free(base); +} + +static void emutls_exit(void) { + if (emutls_mutex) { + DeleteCriticalSection(emutls_mutex); + _aligned_free(emutls_mutex); + emutls_mutex = NULL; + } + if (emutls_tls_index != TLS_OUT_OF_INDEXES) { + emutls_shutdown((emutls_address_array*)TlsGetValue(emutls_tls_index)); + TlsFree(emutls_tls_index); + emutls_tls_index = TLS_OUT_OF_INDEXES; + } +} + +#pragma warning (push) +#pragma warning (disable : 4100) +static BOOL CALLBACK emutls_init(PINIT_ONCE p0, PVOID p1, PVOID *p2) { + emutls_mutex = (LPCRITICAL_SECTION)_aligned_malloc(sizeof(CRITICAL_SECTION), 16); + if (!emutls_mutex) { + win_error(GetLastError(), "_aligned_malloc"); + return FALSE; + } + InitializeCriticalSection(emutls_mutex); + + emutls_tls_index = TlsAlloc(); + if (emutls_tls_index == TLS_OUT_OF_INDEXES) { + emutls_exit(); + win_error(GetLastError(), "TlsAlloc"); + return FALSE; + } + atexit(&emutls_exit); + return TRUE; +} + +static __inline void emutls_init_once(void) { + static INIT_ONCE once; + InitOnceExecuteOnce(&once, emutls_init, NULL, NULL); +} + +static __inline void emutls_lock() { + EnterCriticalSection(emutls_mutex); +} + +static __inline void emutls_unlock() { + LeaveCriticalSection(emutls_mutex); +} + +static __inline void emutls_setspecific(emutls_address_array *value) { + if (TlsSetValue(emutls_tls_index, (LPVOID) value) == 0) + win_abort(GetLastError(), "TlsSetValue"); +} + +static __inline emutls_address_array* emutls_getspecific() { + LPVOID value = TlsGetValue(emutls_tls_index); + if (value == NULL) { + const DWORD err = GetLastError(); + if (err != ERROR_SUCCESS) + win_abort(err, "TlsGetValue"); + } + return (emutls_address_array*) value; +} + +/* Provide atomic load/store functions for emutls_get_index if built with MSVC. + */ +#if !defined(__ATOMIC_RELEASE) + +enum { __ATOMIC_ACQUIRE = 2, __ATOMIC_RELEASE = 3 }; + +static __inline uintptr_t __atomic_load_n(void *ptr, unsigned type) { + assert(type == __ATOMIC_ACQUIRE); +#ifdef _WIN64 + return (uintptr_t) _load_be_u64(ptr); +#else + return (uintptr_t) _load_be_u32(ptr); +#endif +} + +static __inline void __atomic_store_n(void *ptr, uintptr_t val, unsigned type) { + assert(type == __ATOMIC_RELEASE); +#ifdef _WIN64 + _store_be_u64(ptr, val); +#else + _store_be_u32(ptr, val); +#endif +} + +#endif + +#pragma warning (pop) + +#endif + +static size_t emutls_num_object = 0; /* number of allocated TLS objects */ + +/* Free the allocated TLS data + */ +static void emutls_shutdown(emutls_address_array *array) { + if (array) { + uintptr_t i; + for (i = 0; i < array->size; ++i) { + if (array->data[i]) + emutls_memalign_free(array->data[i]); + } + } +} + +/* For every TLS variable xyz, + * there is one __emutls_control variable named __emutls_v.xyz. + * If xyz has non-zero initial value, __emutls_v.xyz's "value" + * will point to __emutls_t.xyz, which has the initial value. + */ +typedef struct __emutls_control { + /* Must use gcc_word here, instead of size_t, to match GCC. When + gcc_word is larger than size_t, the upper extra bits are all + zeros. We can use variables of size_t to operate on size and + align. */ + gcc_word size; /* size of the object in bytes */ + gcc_word align; /* alignment of the object in bytes */ + union { + uintptr_t index; /* data[index-1] is the object address */ + void* address; /* object address, when in single thread env */ + } object; + void* value; /* null or non-zero initial value for the object */ +} __emutls_control; + /* Emulated TLS objects are always allocated at run-time. */ static __inline void *emutls_allocate_object(__emutls_control *control) { /* Use standard C types, check with gcc's emutls.o. */ - typedef unsigned int gcc_pointer __attribute__((mode(pointer))); COMPILE_TIME_ASSERT(sizeof(uintptr_t) == sizeof(gcc_pointer)); COMPILE_TIME_ASSERT(sizeof(uintptr_t) == sizeof(void*)); @@ -93,45 +286,19 @@ static __inline void *emutls_allocate_object(__emutls_control *control) { return base; } -static pthread_mutex_t emutls_mutex = PTHREAD_MUTEX_INITIALIZER; - -static size_t emutls_num_object = 0; /* number of allocated TLS objects */ - -typedef struct emutls_address_array { - uintptr_t size; /* number of elements in the 'data' array */ - void* data[]; -} emutls_address_array; - -static pthread_key_t emutls_pthread_key; - -static void emutls_key_destructor(void* ptr) { - emutls_address_array* array = (emutls_address_array*)ptr; - uintptr_t i; - for (i = 0; i < array->size; ++i) { - if (array->data[i]) - emutls_memalign_free(array->data[i]); - } - free(ptr); -} - -static void emutls_init(void) { - if (pthread_key_create(&emutls_pthread_key, emutls_key_destructor) != 0) - abort(); -} /* Returns control->object.index; set index if not allocated yet. */ static __inline uintptr_t emutls_get_index(__emutls_control *control) { uintptr_t index = __atomic_load_n(&control->object.index, __ATOMIC_ACQUIRE); if (!index) { - static pthread_once_t once = PTHREAD_ONCE_INIT; - pthread_once(&once, emutls_init); - pthread_mutex_lock(&emutls_mutex); + emutls_init_once(); + emutls_lock(); index = control->object.index; if (!index) { index = ++emutls_num_object; __atomic_store_n(&control->object.index, index, __ATOMIC_RELEASE); } - pthread_mutex_unlock(&emutls_mutex); + emutls_unlock(); } return index; } @@ -142,7 +309,7 @@ static __inline void emutls_check_array_set_size(emutls_address_array *array, if (array == NULL) abort(); array->size = size; - pthread_setspecific(emutls_pthread_key, (void*)array); + emutls_setspecific(array); } /* Returns the new 'data' array size, number of elements, @@ -156,22 +323,29 @@ static __inline uintptr_t emutls_new_data_array_size(uintptr_t index) { return ((index + 1 + 15) & ~((uintptr_t)15)) - 1; } +/* Returns the size in bytes required for an emutls_address_array with + * N number of elements for data field. + */ +static __inline uintptr_t emutls_asize(uintptr_t N) { + return N * sizeof(void *) + sizeof(emutls_address_array); +} + /* Returns the thread local emutls_address_array. * Extends its size if necessary to hold address at index. */ static __inline emutls_address_array * emutls_get_address_array(uintptr_t index) { - emutls_address_array* array = pthread_getspecific(emutls_pthread_key); + emutls_address_array* array = emutls_getspecific(); if (array == NULL) { uintptr_t new_size = emutls_new_data_array_size(index); - array = malloc(new_size * sizeof(void *) + sizeof(emutls_address_array)); + array = (emutls_address_array*) malloc(emutls_asize(new_size)); if (array) memset(array->data, 0, new_size * sizeof(void*)); emutls_check_array_set_size(array, new_size); } else if (index > array->size) { uintptr_t orig_size = array->size; uintptr_t new_size = emutls_new_data_array_size(index); - array = realloc(array, new_size * sizeof(void *) + sizeof(emutls_address_array)); + array = (emutls_address_array*) realloc(array, emutls_asize(new_size)); if (array) memset(array->data + orig_size, 0, (new_size - orig_size) * sizeof(void*)); @@ -182,8 +356,8 @@ emutls_get_address_array(uintptr_t index) { void* __emutls_get_address(__emutls_control* control) { uintptr_t index = emutls_get_index(control); - emutls_address_array* array = emutls_get_address_array(index); - if (array->data[index - 1] == NULL) - array->data[index - 1] = emutls_allocate_object(control); - return array->data[index - 1]; + emutls_address_array* array = emutls_get_address_array(index--); + if (array->data[index] == NULL) + array->data[index] = emutls_allocate_object(control); + return array->data[index]; } diff --git a/contrib/compiler-rt/lib/builtins/extendhfsf2.c b/contrib/compiler-rt/lib/builtins/extendhfsf2.c index 27115a4..e7d9fde 100644 --- a/contrib/compiler-rt/lib/builtins/extendhfsf2.c +++ b/contrib/compiler-rt/lib/builtins/extendhfsf2.c @@ -12,8 +12,6 @@ #define DST_SINGLE #include "fp_extend_impl.inc" -ARM_EABI_FNALIAS(h2f, extendhfsf2) - // Use a forwarding definition and noinline to implement a poor man's alias, // as there isn't a good cross-platform way of defining one. COMPILER_RT_ABI NOINLINE float __extendhfsf2(uint16_t a) { @@ -23,3 +21,10 @@ COMPILER_RT_ABI NOINLINE float __extendhfsf2(uint16_t a) { COMPILER_RT_ABI float __gnu_h2f_ieee(uint16_t a) { return __extendhfsf2(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI float __aeabi_h2f(uint16_t a) { + return __extendhfsf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/extendsfdf2.c b/contrib/compiler-rt/lib/builtins/extendsfdf2.c index 7a267c2..b9e7a74 100644 --- a/contrib/compiler-rt/lib/builtins/extendsfdf2.c +++ b/contrib/compiler-rt/lib/builtins/extendsfdf2.c @@ -12,8 +12,13 @@ #define DST_DOUBLE #include "fp_extend_impl.inc" -ARM_EABI_FNALIAS(f2d, extendsfdf2) - COMPILER_RT_ABI double __extendsfdf2(float a) { return __extendXfYf2__(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI double __aeabi_f2d(float a) { + return __extendsfdf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/ffssi2.c b/contrib/compiler-rt/lib/builtins/ffssi2.c new file mode 100644 index 0000000..e5180ef --- /dev/null +++ b/contrib/compiler-rt/lib/builtins/ffssi2.c @@ -0,0 +1,29 @@ +/* ===-- ffssi2.c - Implement __ffssi2 -------------------------------------=== + * + * The LLVM Compiler Infrastructure + * + * This file is dual licensed under the MIT and the University of Illinois Open + * Source Licenses. See LICENSE.TXT for details. + * + * ===----------------------------------------------------------------------=== + * + * This file implements __ffssi2 for the compiler_rt library. + * + * ===----------------------------------------------------------------------=== + */ + +#include "int_lib.h" + +/* Returns: the index of the least significant 1-bit in a, or + * the value zero if a is zero. The least significant bit is index one. + */ + +COMPILER_RT_ABI si_int +__ffssi2(si_int a) +{ + if (a == 0) + { + return 0; + } + return __builtin_ctz(a) + 1; +} diff --git a/contrib/compiler-rt/lib/builtins/fixdfdi.c b/contrib/compiler-rt/lib/builtins/fixdfdi.c index 14283ef..31d76df 100644 --- a/contrib/compiler-rt/lib/builtins/fixdfdi.c +++ b/contrib/compiler-rt/lib/builtins/fixdfdi.c @@ -10,7 +10,6 @@ #define DOUBLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(d2lz, fixdfdi) #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; can set the invalid @@ -44,3 +43,15 @@ __fixdfdi(fp_t a) { } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int +#if defined(__SOFT_FP__) +__aeabi_d2lz(fp_t a) { +#else +__aeabi_d2lz(double a) { +#endif + return __fixdfdi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixdfsi.c b/contrib/compiler-rt/lib/builtins/fixdfsi.c index 704e65b..fc316dc 100644 --- a/contrib/compiler-rt/lib/builtins/fixdfsi.c +++ b/contrib/compiler-rt/lib/builtins/fixdfsi.c @@ -14,9 +14,14 @@ typedef si_int fixint_t; typedef su_int fixuint_t; #include "fp_fixint_impl.inc" -ARM_EABI_FNALIAS(d2iz, fixdfsi) - COMPILER_RT_ABI si_int __fixdfsi(fp_t a) { return __fixint(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI si_int __aeabi_d2iz(fp_t a) { + return __fixdfsi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixsfdi.c b/contrib/compiler-rt/lib/builtins/fixsfdi.c index fab47e2..c434736 100644 --- a/contrib/compiler-rt/lib/builtins/fixsfdi.c +++ b/contrib/compiler-rt/lib/builtins/fixsfdi.c @@ -11,8 +11,6 @@ #define SINGLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(f2lz, fixsfdi) - #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; can set the invalid * flag as a side-effect of computation. @@ -45,3 +43,15 @@ __fixsfdi(fp_t a) { } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int +#if defined(__SOFT_FP__) +__aeabi_f2lz(fp_t a) { +#else +__aeabi_f2lz(float a) { +#endif + return __fixsfdi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixsfsi.c b/contrib/compiler-rt/lib/builtins/fixsfsi.c index f045536..3276df9 100644 --- a/contrib/compiler-rt/lib/builtins/fixsfsi.c +++ b/contrib/compiler-rt/lib/builtins/fixsfsi.c @@ -14,9 +14,14 @@ typedef si_int fixint_t; typedef su_int fixuint_t; #include "fp_fixint_impl.inc" -ARM_EABI_FNALIAS(f2iz, fixsfsi) - COMPILER_RT_ABI si_int __fixsfsi(fp_t a) { return __fixint(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI si_int __aeabi_f2iz(fp_t a) { + return __fixsfsi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixunsdfdi.c b/contrib/compiler-rt/lib/builtins/fixunsdfdi.c index 4b0bc9e1..b734409 100644 --- a/contrib/compiler-rt/lib/builtins/fixunsdfdi.c +++ b/contrib/compiler-rt/lib/builtins/fixunsdfdi.c @@ -11,8 +11,6 @@ #define DOUBLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(d2ulz, fixunsdfdi) - #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; can set the invalid * flag as a side-effect of computation. @@ -42,3 +40,15 @@ __fixunsdfdi(fp_t a) { } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI du_int +#if defined(__SOFT_FP__) +__aeabi_d2ulz(fp_t a) { +#else +__aeabi_d2ulz(double a) { +#endif + return __fixunsdfdi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixunsdfsi.c b/contrib/compiler-rt/lib/builtins/fixunsdfsi.c index 232d342..bb3d8e0 100644 --- a/contrib/compiler-rt/lib/builtins/fixunsdfsi.c +++ b/contrib/compiler-rt/lib/builtins/fixunsdfsi.c @@ -13,9 +13,14 @@ typedef su_int fixuint_t; #include "fp_fixuint_impl.inc" -ARM_EABI_FNALIAS(d2uiz, fixunsdfsi) - COMPILER_RT_ABI su_int __fixunsdfsi(fp_t a) { return __fixuint(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI su_int __aeabi_d2uiz(fp_t a) { + return __fixunsdfsi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixunssfdi.c b/contrib/compiler-rt/lib/builtins/fixunssfdi.c index f8ebab8..5d92245 100644 --- a/contrib/compiler-rt/lib/builtins/fixunssfdi.c +++ b/contrib/compiler-rt/lib/builtins/fixunssfdi.c @@ -11,8 +11,6 @@ #define SINGLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(f2ulz, fixunssfdi) - #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; can set the invalid * flag as a side-effect of computation. @@ -43,3 +41,15 @@ __fixunssfdi(fp_t a) { } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI du_int +#if defined(__SOFT_FP__) +__aeabi_f2ulz(fp_t a) { +#else +__aeabi_f2ulz(float a) { +#endif + return __fixunssfdi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/fixunssfsi.c b/contrib/compiler-rt/lib/builtins/fixunssfsi.c index cc2b05b..91d5e8a 100644 --- a/contrib/compiler-rt/lib/builtins/fixunssfsi.c +++ b/contrib/compiler-rt/lib/builtins/fixunssfsi.c @@ -17,9 +17,14 @@ typedef su_int fixuint_t; #include "fp_fixuint_impl.inc" -ARM_EABI_FNALIAS(f2uiz, fixunssfsi) - COMPILER_RT_ABI su_int __fixunssfsi(fp_t a) { return __fixuint(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI su_int __aeabi_f2uiz(fp_t a) { + return __fixunssfsi(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatdidf.c b/contrib/compiler-rt/lib/builtins/floatdidf.c index 2b023ad..681fece 100644 --- a/contrib/compiler-rt/lib/builtins/floatdidf.c +++ b/contrib/compiler-rt/lib/builtins/floatdidf.c @@ -22,8 +22,6 @@ /* seee eeee eeee mmmm mmmm mmmm mmmm mmmm | mmmm mmmm mmmm mmmm mmmm mmmm mmmm mmmm */ -ARM_EABI_FNALIAS(l2d, floatdidf) - #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; we'll set the inexact flag * as a side-effect of this computation. @@ -105,3 +103,10 @@ __floatdidf(di_int a) return fb.f; } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI double __aeabi_l2d(di_int a) { + return __floatdidf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatdisf.c b/contrib/compiler-rt/lib/builtins/floatdisf.c index 3e47580..dd54816 100644 --- a/contrib/compiler-rt/lib/builtins/floatdisf.c +++ b/contrib/compiler-rt/lib/builtins/floatdisf.c @@ -22,8 +22,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(l2f, floatdisf) - COMPILER_RT_ABI float __floatdisf(di_int a) { @@ -78,3 +76,10 @@ __floatdisf(di_int a) ((su_int)a & 0x007FFFFF); /* mantissa */ return fb.f; } + +#if defined(__ARM_EABI__) +AEABI_RTABI float __aeabi_l2f(di_int a) { + return __floatdisf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatsidf.c b/contrib/compiler-rt/lib/builtins/floatsidf.c index 1cf99b7..2ae395b 100644 --- a/contrib/compiler-rt/lib/builtins/floatsidf.c +++ b/contrib/compiler-rt/lib/builtins/floatsidf.c @@ -18,8 +18,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(i2d, floatsidf) - COMPILER_RT_ABI fp_t __floatsidf(int a) { @@ -51,3 +49,10 @@ __floatsidf(int a) { // Insert the sign bit and return return fromRep(result | sign); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_i2d(int a) { + return __floatsidf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatsisf.c b/contrib/compiler-rt/lib/builtins/floatsisf.c index 467dd1d..08891fc 100644 --- a/contrib/compiler-rt/lib/builtins/floatsisf.c +++ b/contrib/compiler-rt/lib/builtins/floatsisf.c @@ -18,8 +18,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(i2f, floatsisf) - COMPILER_RT_ABI fp_t __floatsisf(int a) { @@ -57,3 +55,10 @@ __floatsisf(int a) { // Insert the sign bit and return return fromRep(result | sign); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_i2f(int a) { + return __floatsisf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatundidf.c b/contrib/compiler-rt/lib/builtins/floatundidf.c index cfd3a7a..6c1a931 100644 --- a/contrib/compiler-rt/lib/builtins/floatundidf.c +++ b/contrib/compiler-rt/lib/builtins/floatundidf.c @@ -22,8 +22,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(ul2d, floatundidf) - #ifndef __SOFT_FP__ /* Support for systems that have hardware floating-point; we'll set the inexact flag * as a side-effect of this computation. @@ -104,3 +102,10 @@ __floatundidf(du_int a) return fb.f; } #endif + +#if defined(__ARM_EABI__) +AEABI_RTABI double __aeabi_ul2d(du_int a) { + return __floatundidf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatundisf.c b/contrib/compiler-rt/lib/builtins/floatundisf.c index 713a44a..86841a7 100644 --- a/contrib/compiler-rt/lib/builtins/floatundisf.c +++ b/contrib/compiler-rt/lib/builtins/floatundisf.c @@ -22,8 +22,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(ul2f, floatundisf) - COMPILER_RT_ABI float __floatundisf(du_int a) { @@ -75,3 +73,10 @@ __floatundisf(du_int a) ((su_int)a & 0x007FFFFF); /* mantissa */ return fb.f; } + +#if defined(__ARM_EABI__) +AEABI_RTABI float __aeabi_ul2f(du_int a) { + return __floatundisf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatunsidf.c b/contrib/compiler-rt/lib/builtins/floatunsidf.c index 445e180..8d48071 100644 --- a/contrib/compiler-rt/lib/builtins/floatunsidf.c +++ b/contrib/compiler-rt/lib/builtins/floatunsidf.c @@ -18,8 +18,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(ui2d, floatunsidf) - COMPILER_RT_ABI fp_t __floatunsidf(unsigned int a) { @@ -40,3 +38,10 @@ __floatunsidf(unsigned int a) { result += (rep_t)(exponent + exponentBias) << significandBits; return fromRep(result); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_ui2d(unsigned int a) { + return __floatunsidf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/floatunsisf.c b/contrib/compiler-rt/lib/builtins/floatunsisf.c index ea6f161..f194c04 100644 --- a/contrib/compiler-rt/lib/builtins/floatunsisf.c +++ b/contrib/compiler-rt/lib/builtins/floatunsisf.c @@ -18,8 +18,6 @@ #include "int_lib.h" -ARM_EABI_FNALIAS(ui2f, floatunsisf) - COMPILER_RT_ABI fp_t __floatunsisf(unsigned int a) { @@ -48,3 +46,10 @@ __floatunsisf(unsigned int a) { result += (rep_t)(exponent + exponentBias) << significandBits; return fromRep(result); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_ui2f(unsigned int a) { + return __floatunsisf(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/int_lib.h b/contrib/compiler-rt/lib/builtins/int_lib.h index 09f27f8..edb040a 100644 --- a/contrib/compiler-rt/lib/builtins/int_lib.h +++ b/contrib/compiler-rt/lib/builtins/int_lib.h @@ -30,22 +30,19 @@ /* ABI macro definitions */ #if __ARM_EABI__ -# define ARM_EABI_FNALIAS(aeabi_name, name) \ - void __aeabi_##aeabi_name() __attribute__((alias("__" #name))); - -# if !defined(__clang__) && defined(__GNUC__) && \ - (__GNUC__ < 4 || __GNUC__ == 4 && __GNUC_MINOR__ < 5) +# if defined(COMPILER_RT_ARMHF_TARGET) || (!defined(__clang__) && \ + defined(__GNUC__) && (__GNUC__ < 4 || __GNUC__ == 4 && __GNUC_MINOR__ < 5)) /* The pcs attribute was introduced in GCC 4.5.0 */ -# define COMPILER_RT_ABI +# define COMPILER_RT_ABI # else -# define COMPILER_RT_ABI __attribute__((pcs("aapcs"))) +# define COMPILER_RT_ABI __attribute__((__pcs__("aapcs"))) # endif - #else -# define ARM_EABI_FNALIAS(aeabi_name, name) # define COMPILER_RT_ABI #endif +#define AEABI_RTABI __attribute__((__pcs__("aapcs"))) + #ifdef _MSC_VER #define ALWAYS_INLINE __forceinline #define NOINLINE __declspec(noinline) diff --git a/contrib/compiler-rt/lib/builtins/int_types.h b/contrib/compiler-rt/lib/builtins/int_types.h index 660385e..a92238c 100644 --- a/contrib/compiler-rt/lib/builtins/int_types.h +++ b/contrib/compiler-rt/lib/builtins/int_types.h @@ -60,9 +60,7 @@ typedef union }s; } udwords; -/* MIPS64 issue: PR 20098 */ -#if (defined(__LP64__) || defined(__wasm__)) && \ - !(defined(__mips__) && defined(__clang__)) +#if (defined(__LP64__) || defined(__wasm__) || defined(__mips64)) #define CRT_HAS_128BIT #endif diff --git a/contrib/compiler-rt/lib/builtins/int_util.c b/contrib/compiler-rt/lib/builtins/int_util.c index 420d1e2..de87410 100644 --- a/contrib/compiler-rt/lib/builtins/int_util.c +++ b/contrib/compiler-rt/lib/builtins/int_util.c @@ -45,6 +45,16 @@ void compilerrt_abort_impl(const char *file, int line, const char *function) { __assert_rtn(function, file, line, "libcompiler_rt abort"); } +#elif __Fuchsia__ + +#ifndef _WIN32 +__attribute__((weak)) +__attribute__((visibility("hidden"))) +#endif +void compilerrt_abort_impl(const char *file, int line, const char *function) { + __builtin_trap(); +} + #else /* Get the system definition of abort() */ diff --git a/contrib/compiler-rt/lib/builtins/lshrdi3.c b/contrib/compiler-rt/lib/builtins/lshrdi3.c index 6b1ea92..becbbef4 100644 --- a/contrib/compiler-rt/lib/builtins/lshrdi3.c +++ b/contrib/compiler-rt/lib/builtins/lshrdi3.c @@ -18,8 +18,6 @@ /* Precondition: 0 <= b < bits_in_dword */ -ARM_EABI_FNALIAS(llsr, lshrdi3) - COMPILER_RT_ABI di_int __lshrdi3(di_int a, si_int b) { @@ -41,3 +39,10 @@ __lshrdi3(di_int a, si_int b) } return result.all; } + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int __aeabi_llsr(di_int a, si_int b) { + return __lshrdi3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/muldf3.c b/contrib/compiler-rt/lib/builtins/muldf3.c index 1eb7338..59a6019 100644 --- a/contrib/compiler-rt/lib/builtins/muldf3.c +++ b/contrib/compiler-rt/lib/builtins/muldf3.c @@ -15,8 +15,13 @@ #define DOUBLE_PRECISION #include "fp_mul_impl.inc" -ARM_EABI_FNALIAS(dmul, muldf3) - COMPILER_RT_ABI fp_t __muldf3(fp_t a, fp_t b) { return __mulXf3__(a, b); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_dmul(fp_t a, fp_t b) { + return __muldf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/muldi3.c b/contrib/compiler-rt/lib/builtins/muldi3.c index 2dae44c..6818a9e 100644 --- a/contrib/compiler-rt/lib/builtins/muldi3.c +++ b/contrib/compiler-rt/lib/builtins/muldi3.c @@ -40,8 +40,6 @@ __muldsi3(su_int a, su_int b) /* Returns: a * b */ -ARM_EABI_FNALIAS(lmul, muldi3) - COMPILER_RT_ABI di_int __muldi3(di_int a, di_int b) { @@ -54,3 +52,10 @@ __muldi3(di_int a, di_int b) r.s.high += x.s.high * y.s.low + x.s.low * y.s.high; return r.all; } + +#if defined(__ARM_EABI__) +AEABI_RTABI di_int __aeabi_lmul(di_int a, di_int b) { + return __muldi3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/mulsf3.c b/contrib/compiler-rt/lib/builtins/mulsf3.c index 478b3bc..f141af1 100644 --- a/contrib/compiler-rt/lib/builtins/mulsf3.c +++ b/contrib/compiler-rt/lib/builtins/mulsf3.c @@ -15,8 +15,13 @@ #define SINGLE_PRECISION #include "fp_mul_impl.inc" -ARM_EABI_FNALIAS(fmul, mulsf3) - COMPILER_RT_ABI fp_t __mulsf3(fp_t a, fp_t b) { return __mulXf3__(a, b); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_fmul(fp_t a, fp_t b) { + return __mulsf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/negdf2.c b/contrib/compiler-rt/lib/builtins/negdf2.c index d634b42..5e2544c 100644 --- a/contrib/compiler-rt/lib/builtins/negdf2.c +++ b/contrib/compiler-rt/lib/builtins/negdf2.c @@ -14,9 +14,14 @@ #define DOUBLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(dneg, negdf2) - COMPILER_RT_ABI fp_t __negdf2(fp_t a) { return fromRep(toRep(a) ^ signBit); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_dneg(fp_t a) { + return __negdf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/negsf2.c b/contrib/compiler-rt/lib/builtins/negsf2.c index 29c17be..f90b343 100644 --- a/contrib/compiler-rt/lib/builtins/negsf2.c +++ b/contrib/compiler-rt/lib/builtins/negsf2.c @@ -14,9 +14,14 @@ #define SINGLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(fneg, negsf2) - COMPILER_RT_ABI fp_t __negsf2(fp_t a) { return fromRep(toRep(a) ^ signBit); } + +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_fneg(fp_t a) { + return __negsf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/os_version_check.c b/contrib/compiler-rt/lib/builtins/os_version_check.c new file mode 100644 index 0000000..74ade2f --- /dev/null +++ b/contrib/compiler-rt/lib/builtins/os_version_check.c @@ -0,0 +1,178 @@ +/* ===-- os_version_check.c - OS version checking -------------------------=== + * + * The LLVM Compiler Infrastructure + * + * This file is dual licensed under the MIT and the University of Illinois Open + * Source Licenses. See LICENSE.TXT for details. + * + * ===----------------------------------------------------------------------=== + * + * This file implements the function __isOSVersionAtLeast, used by + * Objective-C's @available + * + * ===----------------------------------------------------------------------=== + */ + +#ifdef __APPLE__ + +#include <CoreFoundation/CoreFoundation.h> +#include <dispatch/dispatch.h> +#include <TargetConditionals.h> +#include <dlfcn.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +/* These three variables hold the host's OS version. */ +static int32_t GlobalMajor, GlobalMinor, GlobalSubminor; +static dispatch_once_t DispatchOnceCounter; + +/* Find and parse the SystemVersion.plist file. */ +static void parseSystemVersionPList(void *Unused) { + (void)Unused; + /* Load CoreFoundation dynamically */ + const void *NullAllocator = dlsym(RTLD_DEFAULT, "kCFAllocatorNull"); + if (!NullAllocator) + return; + const CFAllocatorRef kCFAllocatorNull = + *(const CFAllocatorRef *)NullAllocator; + typeof(CFDataCreateWithBytesNoCopy) *CFDataCreateWithBytesNoCopyFunc = + (typeof(CFDataCreateWithBytesNoCopy) *)dlsym( + RTLD_DEFAULT, "CFDataCreateWithBytesNoCopy"); + if (!CFDataCreateWithBytesNoCopyFunc) + return; + typeof(CFPropertyListCreateWithData) *CFPropertyListCreateWithDataFunc = + (typeof(CFPropertyListCreateWithData) *)dlsym( + RTLD_DEFAULT, "CFPropertyListCreateWithData"); + /* CFPropertyListCreateWithData was introduced only in macOS 10.6+, so it + * will be NULL on earlier OS versions. */ +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wdeprecated-declarations" + typeof(CFPropertyListCreateFromXMLData) *CFPropertyListCreateFromXMLDataFunc = + (typeof(CFPropertyListCreateFromXMLData) *)dlsym( + RTLD_DEFAULT, "CFPropertyListCreateFromXMLData"); +#pragma clang diagnostic pop + /* CFPropertyListCreateFromXMLDataFunc is deprecated in macOS 10.10, so it + * might be NULL in future OS versions. */ + if (!CFPropertyListCreateWithDataFunc && !CFPropertyListCreateFromXMLDataFunc) + return; + typeof(CFStringCreateWithCStringNoCopy) *CFStringCreateWithCStringNoCopyFunc = + (typeof(CFStringCreateWithCStringNoCopy) *)dlsym( + RTLD_DEFAULT, "CFStringCreateWithCStringNoCopy"); + if (!CFStringCreateWithCStringNoCopyFunc) + return; + typeof(CFDictionaryGetValue) *CFDictionaryGetValueFunc = + (typeof(CFDictionaryGetValue) *)dlsym(RTLD_DEFAULT, + "CFDictionaryGetValue"); + if (!CFDictionaryGetValueFunc) + return; + typeof(CFGetTypeID) *CFGetTypeIDFunc = + (typeof(CFGetTypeID) *)dlsym(RTLD_DEFAULT, "CFGetTypeID"); + if (!CFGetTypeIDFunc) + return; + typeof(CFStringGetTypeID) *CFStringGetTypeIDFunc = + (typeof(CFStringGetTypeID) *)dlsym(RTLD_DEFAULT, "CFStringGetTypeID"); + if (!CFStringGetTypeIDFunc) + return; + typeof(CFStringGetCString) *CFStringGetCStringFunc = + (typeof(CFStringGetCString) *)dlsym(RTLD_DEFAULT, "CFStringGetCString"); + if (!CFStringGetCStringFunc) + return; + typeof(CFRelease) *CFReleaseFunc = + (typeof(CFRelease) *)dlsym(RTLD_DEFAULT, "CFRelease"); + if (!CFReleaseFunc) + return; + + char *PListPath = "/System/Library/CoreServices/SystemVersion.plist"; + +#if TARGET_OS_SIMULATOR + char *PListPathPrefix = getenv("IPHONE_SIMULATOR_ROOT"); + if (!PListPathPrefix) + return; + char FullPath[strlen(PListPathPrefix) + strlen(PListPath) + 1]; + strcpy(FullPath, PListPathPrefix); + strcat(FullPath, PListPath); + PListPath = FullPath; +#endif + FILE *PropertyList = fopen(PListPath, "r"); + if (!PropertyList) + return; + + /* Dynamically allocated stuff. */ + CFDictionaryRef PListRef = NULL; + CFDataRef FileContentsRef = NULL; + UInt8 *PListBuf = NULL; + + fseek(PropertyList, 0, SEEK_END); + long PListFileSize = ftell(PropertyList); + if (PListFileSize < 0) + goto Fail; + rewind(PropertyList); + + PListBuf = malloc((size_t)PListFileSize); + if (!PListBuf) + goto Fail; + + size_t NumRead = fread(PListBuf, 1, (size_t)PListFileSize, PropertyList); + if (NumRead != (size_t)PListFileSize) + goto Fail; + + /* Get the file buffer into CF's format. We pass in a null allocator here * + * because we free PListBuf ourselves */ + FileContentsRef = (*CFDataCreateWithBytesNoCopyFunc)( + NULL, PListBuf, (CFIndex)NumRead, kCFAllocatorNull); + if (!FileContentsRef) + goto Fail; + + if (CFPropertyListCreateWithDataFunc) + PListRef = (*CFPropertyListCreateWithDataFunc)( + NULL, FileContentsRef, kCFPropertyListImmutable, NULL, NULL); + else + PListRef = (*CFPropertyListCreateFromXMLDataFunc)( + NULL, FileContentsRef, kCFPropertyListImmutable, NULL); + if (!PListRef) + goto Fail; + + CFStringRef ProductVersion = (*CFStringCreateWithCStringNoCopyFunc)( + NULL, "ProductVersion", kCFStringEncodingASCII, kCFAllocatorNull); + if (!ProductVersion) + goto Fail; + CFTypeRef OpaqueValue = (*CFDictionaryGetValueFunc)(PListRef, ProductVersion); + (*CFReleaseFunc)(ProductVersion); + if (!OpaqueValue || + (*CFGetTypeIDFunc)(OpaqueValue) != (*CFStringGetTypeIDFunc)()) + goto Fail; + + char VersionStr[32]; + if (!(*CFStringGetCStringFunc)((CFStringRef)OpaqueValue, VersionStr, + sizeof(VersionStr), kCFStringEncodingUTF8)) + goto Fail; + sscanf(VersionStr, "%d.%d.%d", &GlobalMajor, &GlobalMinor, &GlobalSubminor); + +Fail: + if (PListRef) + (*CFReleaseFunc)(PListRef); + if (FileContentsRef) + (*CFReleaseFunc)(FileContentsRef); + free(PListBuf); + fclose(PropertyList); +} + +int32_t __isOSVersionAtLeast(int32_t Major, int32_t Minor, int32_t Subminor) { + /* Populate the global version variables, if they haven't already. */ + dispatch_once_f(&DispatchOnceCounter, NULL, parseSystemVersionPList); + + if (Major < GlobalMajor) return 1; + if (Major > GlobalMajor) return 0; + if (Minor < GlobalMinor) return 1; + if (Minor > GlobalMinor) return 0; + return Subminor <= GlobalSubminor; +} + +#else + +/* Silence an empty translation unit warning. */ +typedef int unused; + +#endif diff --git a/contrib/compiler-rt/lib/builtins/subdf3.c b/contrib/compiler-rt/lib/builtins/subdf3.c index 7a79e5e..38340df 100644 --- a/contrib/compiler-rt/lib/builtins/subdf3.c +++ b/contrib/compiler-rt/lib/builtins/subdf3.c @@ -15,11 +15,15 @@ #define DOUBLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(dsub, subdf3) - // Subtraction; flip the sign bit of b and add. COMPILER_RT_ABI fp_t __subdf3(fp_t a, fp_t b) { return __adddf3(a, fromRep(toRep(b) ^ signBit)); } +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_dsub(fp_t a, fp_t b) { + return __subdf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/subsf3.c b/contrib/compiler-rt/lib/builtins/subsf3.c index c3b8514..34276b1 100644 --- a/contrib/compiler-rt/lib/builtins/subsf3.c +++ b/contrib/compiler-rt/lib/builtins/subsf3.c @@ -15,11 +15,15 @@ #define SINGLE_PRECISION #include "fp_lib.h" -ARM_EABI_FNALIAS(fsub, subsf3) - // Subtraction; flip the sign bit of b and add. COMPILER_RT_ABI fp_t __subsf3(fp_t a, fp_t b) { return __addsf3(a, fromRep(toRep(b) ^ signBit)); } +#if defined(__ARM_EABI__) +AEABI_RTABI fp_t __aeabi_fsub(fp_t a, fp_t b) { + return __subsf3(a, b); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/truncdfhf2.c b/contrib/compiler-rt/lib/builtins/truncdfhf2.c index 17195cd..4bb71aa 100644 --- a/contrib/compiler-rt/lib/builtins/truncdfhf2.c +++ b/contrib/compiler-rt/lib/builtins/truncdfhf2.c @@ -11,8 +11,13 @@ #define DST_HALF #include "fp_trunc_impl.inc" -ARM_EABI_FNALIAS(d2h, truncdfhf2) - COMPILER_RT_ABI uint16_t __truncdfhf2(double a) { return __truncXfYf2__(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI uint16_t __aeabi_d2h(double a) { + return __truncdfhf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/truncdfsf2.c b/contrib/compiler-rt/lib/builtins/truncdfsf2.c index 46ec11d..8bf58bb 100644 --- a/contrib/compiler-rt/lib/builtins/truncdfsf2.c +++ b/contrib/compiler-rt/lib/builtins/truncdfsf2.c @@ -11,8 +11,13 @@ #define DST_SINGLE #include "fp_trunc_impl.inc" -ARM_EABI_FNALIAS(d2f, truncdfsf2) - COMPILER_RT_ABI float __truncdfsf2(double a) { return __truncXfYf2__(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI float __aeabi_d2f(double a) { + return __truncdfsf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/truncsfhf2.c b/contrib/compiler-rt/lib/builtins/truncsfhf2.c index 9d61895..f6ce1fa 100644 --- a/contrib/compiler-rt/lib/builtins/truncsfhf2.c +++ b/contrib/compiler-rt/lib/builtins/truncsfhf2.c @@ -11,8 +11,6 @@ #define DST_HALF #include "fp_trunc_impl.inc" -ARM_EABI_FNALIAS(f2h, truncsfhf2) - // Use a forwarding definition and noinline to implement a poor man's alias, // as there isn't a good cross-platform way of defining one. COMPILER_RT_ABI NOINLINE uint16_t __truncsfhf2(float a) { @@ -22,3 +20,10 @@ COMPILER_RT_ABI NOINLINE uint16_t __truncsfhf2(float a) { COMPILER_RT_ABI uint16_t __gnu_f2h_ieee(float a) { return __truncsfhf2(a); } + +#if defined(__ARM_EABI__) +AEABI_RTABI uint16_t __aeabi_f2h(float a) { + return __truncsfhf2(a); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/udivsi3.c b/contrib/compiler-rt/lib/builtins/udivsi3.c index 5d0140c..8eccf10 100644 --- a/contrib/compiler-rt/lib/builtins/udivsi3.c +++ b/contrib/compiler-rt/lib/builtins/udivsi3.c @@ -18,8 +18,6 @@ /* Translated from Figure 3-40 of The PowerPC Compiler Writer's Guide */ -ARM_EABI_FNALIAS(uidiv, udivsi3) - /* This function should not call __divsi3! */ COMPILER_RT_ABI su_int __udivsi3(su_int n, su_int d) @@ -64,3 +62,10 @@ __udivsi3(su_int n, su_int d) q = (q << 1) | carry; return q; } + +#if defined(__ARM_EABI__) +AEABI_RTABI su_int __aeabi_uidiv(su_int n, su_int d) { + return __udivsi3(n, d); +} +#endif + diff --git a/contrib/compiler-rt/lib/builtins/x86_64/floatdidf.c b/contrib/compiler-rt/lib/builtins/x86_64/floatdidf.c index 388404e..dead0ed 100644 --- a/contrib/compiler-rt/lib/builtins/x86_64/floatdidf.c +++ b/contrib/compiler-rt/lib/builtins/x86_64/floatdidf.c @@ -4,7 +4,7 @@ /* double __floatdidf(di_int a); */ -#ifdef __x86_64__ +#if defined(__x86_64__) || defined(_M_X64) #include "../int_lib.h" diff --git a/contrib/compiler-rt/lib/builtins/x86_64/floatdisf.c b/contrib/compiler-rt/lib/builtins/x86_64/floatdisf.c index 96c3728..99d5621 100644 --- a/contrib/compiler-rt/lib/builtins/x86_64/floatdisf.c +++ b/contrib/compiler-rt/lib/builtins/x86_64/floatdisf.c @@ -2,7 +2,7 @@ * License. See LICENSE.TXT for details. */ -#ifdef __x86_64__ +#if defined(__x86_64__) || defined(_M_X64) #include "../int_lib.h" diff --git a/contrib/compiler-rt/lib/cfi/cfi.cc b/contrib/compiler-rt/lib/cfi/cfi.cc index d463ca8..f720230 100644 --- a/contrib/compiler-rt/lib/cfi/cfi.cc +++ b/contrib/compiler-rt/lib/cfi/cfi.cc @@ -188,12 +188,14 @@ uptr find_cfi_check_in_dso(dl_phdr_info *info) { } } if (!dynamic) return 0; - uptr strtab = 0, symtab = 0; + uptr strtab = 0, symtab = 0, strsz = 0; for (const ElfW(Dyn) *p = dynamic; p->d_tag != PT_NULL; ++p) { if (p->d_tag == DT_SYMTAB) symtab = p->d_un.d_ptr; else if (p->d_tag == DT_STRTAB) strtab = p->d_un.d_ptr; + else if (p->d_tag == DT_STRSZ) + strsz = p->d_un.d_ptr; } if (symtab > strtab) { @@ -209,7 +211,8 @@ uptr find_cfi_check_in_dso(dl_phdr_info *info) { if (phdr->p_type == PT_LOAD) { uptr beg = info->dlpi_addr + phdr->p_vaddr; uptr end = beg + phdr->p_memsz; - if (strtab >= beg && strtab < end && symtab >= beg && symtab < end) + if (strtab >= beg && strtab + strsz < end && symtab >= beg && + symtab < end) break; } } @@ -222,9 +225,14 @@ uptr find_cfi_check_in_dso(dl_phdr_info *info) { for (const ElfW(Sym) *p = (const ElfW(Sym) *)symtab; (ElfW(Addr))p < strtab; ++p) { + // There is no reliable way to find the end of the symbol table. In + // lld-produces files, there are other sections between symtab and strtab. + // Stop looking when the symbol name is not inside strtab. + if (p->st_name >= strsz) break; char *name = (char*)(strtab + p->st_name); if (strcmp(name, "__cfi_check") == 0) { - assert(p->st_info == ELF32_ST_INFO(STB_GLOBAL, STT_FUNC)); + assert(p->st_info == ELF32_ST_INFO(STB_GLOBAL, STT_FUNC) || + p->st_info == ELF32_ST_INFO(STB_WEAK, STT_FUNC)); uptr addr = info->dlpi_addr + p->st_value; return addr; } diff --git a/contrib/compiler-rt/lib/cfi/cfi_blacklist.txt b/contrib/compiler-rt/lib/cfi/cfi_blacklist.txt index 1f0eeb3..cc111be 100644 --- a/contrib/compiler-rt/lib/cfi/cfi_blacklist.txt +++ b/contrib/compiler-rt/lib/cfi/cfi_blacklist.txt @@ -24,3 +24,8 @@ fun:_ZNSt3__19addressof* # Windows C++ stdlib headers that contain bad unrelated casts. src:*xmemory0 src:*xstddef + +# std::_Sp_counted_ptr_inplace::_Sp_counted_ptr_inplace() (libstdc++). +# This ctor is used by std::make_shared and needs to cast to uninitialized T* +# in order to call std::allocator_traits<T>::construct. +fun:_ZNSt23_Sp_counted_ptr_inplace* diff --git a/contrib/compiler-rt/lib/dfsan/done_abilist.txt b/contrib/compiler-rt/lib/dfsan/done_abilist.txt index a00dc54..a560cd7 100644 --- a/contrib/compiler-rt/lib/dfsan/done_abilist.txt +++ b/contrib/compiler-rt/lib/dfsan/done_abilist.txt @@ -285,24 +285,8 @@ fun:__sanitizer_cov_module_init=uninstrumented fun:__sanitizer_cov_module_init=discard fun:__sanitizer_cov_with_check=uninstrumented fun:__sanitizer_cov_with_check=discard -fun:__sanitizer_cov_indir_call16=uninstrumented -fun:__sanitizer_cov_indir_call16=discard -fun:__sanitizer_cov_indir_call16=uninstrumented -fun:__sanitizer_cov_indir_call16=discard -fun:__sanitizer_reset_coverage=uninstrumented -fun:__sanitizer_reset_coverage=discard fun:__sanitizer_set_death_callback=uninstrumented fun:__sanitizer_set_death_callback=discard -fun:__sanitizer_get_coverage_guards=uninstrumented -fun:__sanitizer_get_coverage_guards=discard -fun:__sanitizer_get_number_of_counters=uninstrumented -fun:__sanitizer_get_number_of_counters=discard -fun:__sanitizer_update_counter_bitset_and_clear_counters=uninstrumented -fun:__sanitizer_update_counter_bitset_and_clear_counters=discard -fun:__sanitizer_get_total_unique_coverage=uninstrumented -fun:__sanitizer_get_total_unique_coverage=discard -fun:__sanitizer_get_total_unique_coverage=uninstrumented -fun:__sanitizer_get_total_unique_coverage=discard fun:__sanitizer_update_counter_bitset_and_clear_counters=uninstrumented fun:__sanitizer_update_counter_bitset_and_clear_counters=discard diff --git a/contrib/compiler-rt/lib/esan/esan_interceptors.cpp b/contrib/compiler-rt/lib/esan/esan_interceptors.cpp index 9ae5482..62fa13c 100644 --- a/contrib/compiler-rt/lib/esan/esan_interceptors.cpp +++ b/contrib/compiler-rt/lib/esan/esan_interceptors.cpp @@ -31,6 +31,8 @@ using namespace __esan; // NOLINT // Get the per-platform defines for what is possible to intercept #include "sanitizer_common/sanitizer_platform_interceptors.h" +DECLARE_REAL_AND_INTERCEPTOR(void *, malloc, uptr) + // TODO(bruening): tsan disables several interceptors (getpwent, etc.) claiming // that interception is a perf hit: should we do the same? @@ -304,20 +306,6 @@ INTERCEPTOR(int, unlink, char *path) { return REAL(unlink)(path); } -INTERCEPTOR(uptr, fread, void *ptr, uptr size, uptr nmemb, void *f) { - void *ctx; - COMMON_INTERCEPTOR_ENTER(ctx, fread, ptr, size, nmemb, f); - COMMON_INTERCEPTOR_WRITE_RANGE(ctx, ptr, size * nmemb); - return REAL(fread)(ptr, size, nmemb, f); -} - -INTERCEPTOR(uptr, fwrite, const void *p, uptr size, uptr nmemb, void *f) { - void *ctx; - COMMON_INTERCEPTOR_ENTER(ctx, fwrite, p, size, nmemb, f); - COMMON_INTERCEPTOR_READ_RANGE(ctx, p, size * nmemb); - return REAL(fwrite)(p, size, nmemb, f); -} - INTERCEPTOR(int, puts, const char *s) { void *ctx; COMMON_INTERCEPTOR_ENTER(ctx, puts, s); diff --git a/contrib/compiler-rt/lib/esan/esan_sideline_linux.cpp b/contrib/compiler-rt/lib/esan/esan_sideline_linux.cpp index d04f5909..bc272df 100644 --- a/contrib/compiler-rt/lib/esan/esan_sideline_linux.cpp +++ b/contrib/compiler-rt/lib/esan/esan_sideline_linux.cpp @@ -70,7 +70,7 @@ int SidelineThread::runSideline(void *Arg) { // Set up a signal handler on an alternate stack for safety. InternalScopedBuffer<char> StackMap(SigAltStackSize); - struct sigaltstack SigAltStack; + stack_t SigAltStack; SigAltStack.ss_sp = StackMap.data(); SigAltStack.ss_size = SigAltStackSize; SigAltStack.ss_flags = 0; diff --git a/contrib/compiler-rt/lib/esan/working_set.cpp b/contrib/compiler-rt/lib/esan/working_set.cpp index f391119..e56902c 100644 --- a/contrib/compiler-rt/lib/esan/working_set.cpp +++ b/contrib/compiler-rt/lib/esan/working_set.cpp @@ -160,15 +160,16 @@ static u32 countAndClearShadowValues(u32 BitIdx, uptr ShadowStart, static u32 computeWorkingSizeAndReset(u32 BitIdx) { u32 WorkingSetSize = 0; MemoryMappingLayout MemIter(true/*cache*/); - uptr Start, End, Prot; - while (MemIter.Next(&Start, &End, nullptr/*offs*/, nullptr/*file*/, - 0/*file size*/, &Prot)) { - VPrintf(4, "%s: considering %p-%p app=%d shadow=%d prot=%u\n", - __FUNCTION__, Start, End, Prot, isAppMem(Start), - isShadowMem(Start)); - if (isShadowMem(Start) && (Prot & MemoryMappingLayout::kProtectionWrite)) { - VPrintf(3, "%s: walking %p-%p\n", __FUNCTION__, Start, End); - WorkingSetSize += countAndClearShadowValues(BitIdx, Start, End); + MemoryMappedSegment Segment; + while (MemIter.Next(&Segment)) { + VPrintf(4, "%s: considering %p-%p app=%d shadow=%d prot=%u\n", __FUNCTION__, + Segment.start, Segment.end, Segment.protection, + isAppMem(Segment.start), isShadowMem(Segment.start)); + if (isShadowMem(Segment.start) && Segment.IsWritable()) { + VPrintf(3, "%s: walking %p-%p\n", __FUNCTION__, Segment.start, + Segment.end); + WorkingSetSize += + countAndClearShadowValues(BitIdx, Segment.start, Segment.end); } } return WorkingSetSize; diff --git a/contrib/compiler-rt/lib/interception/interception_win.cc b/contrib/compiler-rt/lib/interception/interception_win.cc index 91abecf..b2902d5 100644 --- a/contrib/compiler-rt/lib/interception/interception_win.cc +++ b/contrib/compiler-rt/lib/interception/interception_win.cc @@ -477,7 +477,7 @@ static size_t GetInstructionSize(uptr address, size_t* rel_offset = nullptr) { switch (*(u8*)address) { case 0xA1: // A1 XX XX XX XX XX XX XX XX : // movabs eax, dword ptr ds:[XXXXXXXX] - return 8; + return 9; } switch (*(u16*)address) { @@ -495,6 +495,11 @@ static size_t GetInstructionSize(uptr address, size_t* rel_offset = nullptr) { case 0x5741: // push r15 case 0x9066: // Two-byte NOP return 2; + + case 0x058B: // 8B 05 XX XX XX XX : mov eax, dword ptr [XX XX XX XX] + if (rel_offset) + *rel_offset = 2; + return 6; } switch (0x00FFFFFF & *(u32*)address) { @@ -878,6 +883,8 @@ uptr InternalGetProcAddress(void *module, const char *func_name) { IMAGE_DATA_DIRECTORY *export_directory = &headers->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT]; + if (export_directory->Size == 0) + return 0; RVAPtr<IMAGE_EXPORT_DIRECTORY> exports(module, export_directory->VirtualAddress); RVAPtr<DWORD> functions(module, exports->AddressOfFunctions); diff --git a/contrib/compiler-rt/lib/lsan/lsan.cc b/contrib/compiler-rt/lib/lsan/lsan.cc index c7c3429..6c4767d 100644 --- a/contrib/compiler-rt/lib/lsan/lsan.cc +++ b/contrib/compiler-rt/lib/lsan/lsan.cc @@ -76,6 +76,7 @@ extern "C" void __lsan_init() { InitializeFlags(); InitCommonLsan(); InitializeAllocator(); + ReplaceSystemMalloc(); InitTlsSize(); InitializeInterceptors(); InitializeThreadRegistry(); diff --git a/contrib/compiler-rt/lib/lsan/lsan.h b/contrib/compiler-rt/lib/lsan/lsan.h index ec5eb93..7446e95 100644 --- a/contrib/compiler-rt/lib/lsan/lsan.h +++ b/contrib/compiler-rt/lib/lsan/lsan.h @@ -38,9 +38,18 @@ GET_STACK_TRACE(__sanitizer::common_flags()->malloc_context_size, \ common_flags()->fast_unwind_on_malloc) +#define GET_STACK_TRACE_THREAD GET_STACK_TRACE(kStackTraceMax, true) + namespace __lsan { void InitializeInterceptors(); +void ReplaceSystemMalloc(); + +#define ENSURE_LSAN_INITED do { \ + CHECK(!lsan_init_is_running); \ + if (!lsan_inited) \ + __lsan_init(); \ +} while (0) } // namespace __lsan diff --git a/contrib/compiler-rt/lib/lsan/lsan_allocator.cc b/contrib/compiler-rt/lib/lsan/lsan_allocator.cc index c805a39..2df58b4 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_allocator.cc +++ b/contrib/compiler-rt/lib/lsan/lsan_allocator.cc @@ -15,7 +15,9 @@ #include "lsan_allocator.h" #include "sanitizer_common/sanitizer_allocator.h" +#include "sanitizer_common/sanitizer_allocator_checks.h" #include "sanitizer_common/sanitizer_allocator_interface.h" +#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_internal_defs.h" #include "sanitizer_common/sanitizer_stackdepot.h" #include "sanitizer_common/sanitizer_stacktrace.h" @@ -24,53 +26,27 @@ extern "C" void *memset(void *ptr, int value, uptr num); namespace __lsan { - -struct ChunkMetadata { - u8 allocated : 8; // Must be first. - ChunkTag tag : 2; - uptr requested_size : 54; - u32 stack_trace_id; -}; - -#if defined(__mips64) || defined(__aarch64__) +#if defined(__i386__) || defined(__arm__) +static const uptr kMaxAllowedMallocSize = 1UL << 30; +#elif defined(__mips64) || defined(__aarch64__) static const uptr kMaxAllowedMallocSize = 4UL << 30; -static const uptr kRegionSizeLog = 20; -static const uptr kNumRegions = SANITIZER_MMAP_RANGE_SIZE >> kRegionSizeLog; -typedef TwoLevelByteMap<(kNumRegions >> 12), 1 << 12> ByteMap; -typedef CompactSizeClassMap SizeClassMap; -typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, - sizeof(ChunkMetadata), SizeClassMap, kRegionSizeLog, ByteMap> - PrimaryAllocator; #else static const uptr kMaxAllowedMallocSize = 8UL << 30; - -struct AP64 { // Allocator64 parameters. Deliberately using a short name. - static const uptr kSpaceBeg = 0x600000000000ULL; - static const uptr kSpaceSize = 0x40000000000ULL; // 4T. - static const uptr kMetadataSize = sizeof(ChunkMetadata); - typedef DefaultSizeClassMap SizeClassMap; - typedef NoOpMapUnmapCallback MapUnmapCallback; - static const uptr kFlags = 0; -}; - -typedef SizeClassAllocator64<AP64> PrimaryAllocator; #endif -typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache; typedef LargeMmapAllocator<> SecondaryAllocator; typedef CombinedAllocator<PrimaryAllocator, AllocatorCache, SecondaryAllocator> Allocator; static Allocator allocator; -static THREADLOCAL AllocatorCache cache; void InitializeAllocator() { + SetAllocatorMayReturnNull(common_flags()->allocator_may_return_null); allocator.InitLinkerInitialized( - common_flags()->allocator_may_return_null, common_flags()->allocator_release_to_os_interval_ms); } void AllocatorThreadFinish() { - allocator.SwallowCache(&cache); + allocator.SwallowCache(GetAllocatorCache()); } static ChunkMetadata *Metadata(const void *p) { @@ -100,9 +76,9 @@ void *Allocate(const StackTrace &stack, uptr size, uptr alignment, size = 1; if (size > kMaxAllowedMallocSize) { Report("WARNING: LeakSanitizer failed to allocate %zu bytes\n", size); - return nullptr; + return Allocator::FailureHandler::OnBadRequest(); } - void *p = allocator.Allocate(&cache, size, alignment, false); + void *p = allocator.Allocate(GetAllocatorCache(), size, alignment); // Do not rely on the allocator to clear the memory (it's slow). if (cleared && allocator.FromPrimary(p)) memset(p, 0, size); @@ -112,11 +88,18 @@ void *Allocate(const StackTrace &stack, uptr size, uptr alignment, return p; } +static void *Calloc(uptr nmemb, uptr size, const StackTrace &stack) { + if (UNLIKELY(CheckForCallocOverflow(size, nmemb))) + return Allocator::FailureHandler::OnBadRequest(); + size *= nmemb; + return Allocate(stack, size, 1, true); +} + void Deallocate(void *p) { if (&__sanitizer_free_hook) __sanitizer_free_hook(p); RunFreeHooks(p); RegisterDeallocation(p); - allocator.Deallocate(&cache, p); + allocator.Deallocate(GetAllocatorCache(), p); } void *Reallocate(const StackTrace &stack, void *p, uptr new_size, @@ -124,17 +107,17 @@ void *Reallocate(const StackTrace &stack, void *p, uptr new_size, RegisterDeallocation(p); if (new_size > kMaxAllowedMallocSize) { Report("WARNING: LeakSanitizer failed to allocate %zu bytes\n", new_size); - allocator.Deallocate(&cache, p); - return nullptr; + allocator.Deallocate(GetAllocatorCache(), p); + return Allocator::FailureHandler::OnBadRequest(); } - p = allocator.Reallocate(&cache, p, new_size, alignment); + p = allocator.Reallocate(GetAllocatorCache(), p, new_size, alignment); RegisterAllocation(stack, p, new_size); return p; } void GetAllocatorCacheRange(uptr *begin, uptr *end) { - *begin = (uptr)&cache; - *end = *begin + sizeof(cache); + *begin = (uptr)GetAllocatorCache(); + *end = *begin + sizeof(AllocatorCache); } uptr GetMallocUsableSize(const void *p) { @@ -143,6 +126,39 @@ uptr GetMallocUsableSize(const void *p) { return m->requested_size; } +void *lsan_memalign(uptr alignment, uptr size, const StackTrace &stack) { + if (UNLIKELY(!IsPowerOfTwo(alignment))) { + errno = errno_EINVAL; + return Allocator::FailureHandler::OnBadRequest(); + } + return SetErrnoOnNull(Allocate(stack, size, alignment, kAlwaysClearMemory)); +} + +void *lsan_malloc(uptr size, const StackTrace &stack) { + return SetErrnoOnNull(Allocate(stack, size, 1, kAlwaysClearMemory)); +} + +void lsan_free(void *p) { + Deallocate(p); +} + +void *lsan_realloc(void *p, uptr size, const StackTrace &stack) { + return SetErrnoOnNull(Reallocate(stack, p, size, 1)); +} + +void *lsan_calloc(uptr nmemb, uptr size, const StackTrace &stack) { + return SetErrnoOnNull(Calloc(nmemb, size, stack)); +} + +void *lsan_valloc(uptr size, const StackTrace &stack) { + return SetErrnoOnNull( + Allocate(stack, size, GetPageSizeCached(), kAlwaysClearMemory)); +} + +uptr lsan_mz_size(const void *p) { + return GetMallocUsableSize(p); +} + ///// Interface to the common LSan module. ///// void LockAllocator() { diff --git a/contrib/compiler-rt/lib/lsan/lsan_allocator.h b/contrib/compiler-rt/lib/lsan/lsan_allocator.h index f564601..5a0d94c 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_allocator.h +++ b/contrib/compiler-rt/lib/lsan/lsan_allocator.h @@ -15,8 +15,10 @@ #ifndef LSAN_ALLOCATOR_H #define LSAN_ALLOCATOR_H +#include "sanitizer_common/sanitizer_allocator.h" #include "sanitizer_common/sanitizer_common.h" #include "sanitizer_common/sanitizer_internal_defs.h" +#include "lsan_common.h" namespace __lsan { @@ -34,6 +36,61 @@ void GetAllocatorCacheRange(uptr *begin, uptr *end); void AllocatorThreadFinish(); void InitializeAllocator(); +const bool kAlwaysClearMemory = true; + +struct ChunkMetadata { + u8 allocated : 8; // Must be first. + ChunkTag tag : 2; +#if SANITIZER_WORDSIZE == 64 + uptr requested_size : 54; +#else + uptr requested_size : 32; + uptr padding : 22; +#endif + u32 stack_trace_id; +}; + +#if defined(__mips64) || defined(__aarch64__) || defined(__i386__) || \ + defined(__arm__) +static const uptr kRegionSizeLog = 20; +static const uptr kNumRegions = SANITIZER_MMAP_RANGE_SIZE >> kRegionSizeLog; +typedef TwoLevelByteMap<(kNumRegions >> 12), 1 << 12> ByteMap; + +struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = sizeof(ChunkMetadata); + typedef __sanitizer::CompactSizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = __lsan::kRegionSizeLog; + typedef __lsan::ByteMap ByteMap; + typedef NoOpMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; +}; +typedef SizeClassAllocator32<AP32> PrimaryAllocator; +#elif defined(__x86_64__) || defined(__powerpc64__) +struct AP64 { // Allocator64 parameters. Deliberately using a short name. + static const uptr kSpaceBeg = 0x600000000000ULL; + static const uptr kSpaceSize = 0x40000000000ULL; // 4T. + static const uptr kMetadataSize = sizeof(ChunkMetadata); + typedef DefaultSizeClassMap SizeClassMap; + typedef NoOpMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; +}; + +typedef SizeClassAllocator64<AP64> PrimaryAllocator; +#endif +typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache; + +AllocatorCache *GetAllocatorCache(); + +void *lsan_memalign(uptr alignment, uptr size, const StackTrace &stack); +void *lsan_malloc(uptr size, const StackTrace &stack); +void lsan_free(void *p); +void *lsan_realloc(void *p, uptr size, const StackTrace &stack); +void *lsan_calloc(uptr nmemb, uptr size, const StackTrace &stack); +void *lsan_valloc(uptr size, const StackTrace &stack); +uptr lsan_mz_size(const void *p); + } // namespace __lsan #endif // LSAN_ALLOCATOR_H diff --git a/contrib/compiler-rt/lib/lsan/lsan_common.cc b/contrib/compiler-rt/lib/lsan/lsan_common.cc index f055452..c121e6a 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_common.cc +++ b/contrib/compiler-rt/lib/lsan/lsan_common.cc @@ -32,20 +32,15 @@ namespace __lsan { // also to protect the global list of root regions. BlockingMutex global_mutex(LINKER_INITIALIZED); -__attribute__((tls_model("initial-exec"))) -THREADLOCAL int disable_counter; -bool DisabledInThisThread() { return disable_counter > 0; } -void DisableInThisThread() { disable_counter++; } -void EnableInThisThread() { - if (!disable_counter && common_flags()->detect_leaks) { +Flags lsan_flags; + +void DisableCounterUnderflow() { + if (common_flags()->detect_leaks) { Report("Unmatched call to __lsan_enable().\n"); Die(); } - disable_counter--; } -Flags lsan_flags; - void Flags::SetDefaults() { #define LSAN_FLAG(Type, Name, DefaultValue, Description) Name = DefaultValue; #include "lsan_flags.inc" @@ -73,6 +68,19 @@ ALIGNED(64) static char suppression_placeholder[sizeof(SuppressionContext)]; static SuppressionContext *suppression_ctx = nullptr; static const char kSuppressionLeak[] = "leak"; static const char *kSuppressionTypes[] = { kSuppressionLeak }; +static const char kStdSuppressions[] = +#if SANITIZER_SUPPRESS_LEAK_ON_PTHREAD_EXIT + // For more details refer to the SANITIZER_SUPPRESS_LEAK_ON_PTHREAD_EXIT + // definition. + "leak:*pthread_exit*\n" +#endif // SANITIZER_SUPPRESS_LEAK_ON_PTHREAD_EXIT +#if SANITIZER_MAC + // For Darwin and os_log/os_trace: https://reviews.llvm.org/D35173 + "leak:*_os_trace*\n" +#endif + // TLS leak in some glibc versions, described in + // https://sourceware.org/bugzilla/show_bug.cgi?id=12650. + "leak:*tls_get_addr*\n"; void InitializeSuppressions() { CHECK_EQ(nullptr, suppression_ctx); @@ -81,6 +89,7 @@ void InitializeSuppressions() { suppression_ctx->ParseFromFile(flags()->suppressions); if (&__lsan_default_suppressions) suppression_ctx->Parse(__lsan_default_suppressions()); + suppression_ctx->Parse(kStdSuppressions); } static SuppressionContext *GetSuppressionContext() { @@ -88,12 +97,9 @@ static SuppressionContext *GetSuppressionContext() { return suppression_ctx; } -struct RootRegion { - const void *begin; - uptr size; -}; +static InternalMmapVector<RootRegion> *root_regions; -InternalMmapVector<RootRegion> *root_regions; +InternalMmapVector<RootRegion> const *GetRootRegions() { return root_regions; } void InitializeRootRegions() { CHECK(!root_regions); @@ -180,6 +186,23 @@ void ScanRangeForPointers(uptr begin, uptr end, } } +// Scans a global range for pointers +void ScanGlobalRange(uptr begin, uptr end, Frontier *frontier) { + uptr allocator_begin = 0, allocator_end = 0; + GetAllocatorGlobalRange(&allocator_begin, &allocator_end); + if (begin <= allocator_begin && allocator_begin < end) { + CHECK_LE(allocator_begin, allocator_end); + CHECK_LE(allocator_end, end); + if (begin < allocator_begin) + ScanRangeForPointers(begin, allocator_begin, frontier, "GLOBAL", + kReachable); + if (allocator_end < end) + ScanRangeForPointers(allocator_end, end, frontier, "GLOBAL", kReachable); + } else { + ScanRangeForPointers(begin, end, frontier, "GLOBAL", kReachable); + } +} + void ForEachExtraStackRangeCb(uptr begin, uptr end, void* arg) { Frontier *frontier = reinterpret_cast<Frontier *>(arg); ScanRangeForPointers(begin, end, frontier, "FAKE STACK", kReachable); @@ -188,11 +211,11 @@ void ForEachExtraStackRangeCb(uptr begin, uptr end, void* arg) { // Scans thread data (stacks and TLS) for heap pointers. static void ProcessThreads(SuspendedThreadsList const &suspended_threads, Frontier *frontier) { - InternalScopedBuffer<uptr> registers(SuspendedThreadsList::RegisterCount()); + InternalScopedBuffer<uptr> registers(suspended_threads.RegisterCount()); uptr registers_begin = reinterpret_cast<uptr>(registers.data()); uptr registers_end = registers_begin + registers.size(); - for (uptr i = 0; i < suspended_threads.thread_count(); i++) { - uptr os_id = static_cast<uptr>(suspended_threads.GetThreadID(i)); + for (uptr i = 0; i < suspended_threads.ThreadCount(); i++) { + tid_t os_id = static_cast<tid_t>(suspended_threads.GetThreadID(i)); LOG_THREADS("Processing thread %d.\n", os_id); uptr stack_begin, stack_end, tls_begin, tls_end, cache_begin, cache_end; DTLS *dtls; @@ -206,11 +229,13 @@ static void ProcessThreads(SuspendedThreadsList const &suspended_threads, continue; } uptr sp; - bool have_registers = - (suspended_threads.GetRegistersAndSP(i, registers.data(), &sp) == 0); - if (!have_registers) { - Report("Unable to get registers from thread %d.\n"); - // If unable to get SP, consider the entire stack to be reachable. + PtraceRegistersStatus have_registers = + suspended_threads.GetRegistersAndSP(i, registers.data(), &sp); + if (have_registers != REGISTERS_AVAILABLE) { + Report("Unable to get registers from thread %d.\n", os_id); + // If unable to get SP, consider the entire stack to be reachable unless + // GetRegistersAndSP failed with ESRCH. + if (have_registers == REGISTERS_UNAVAILABLE_FATAL) continue; sp = stack_begin; } @@ -244,21 +269,23 @@ static void ProcessThreads(SuspendedThreadsList const &suspended_threads, } if (flags()->use_tls) { - LOG_THREADS("TLS at %p-%p.\n", tls_begin, tls_end); - if (cache_begin == cache_end) { - ScanRangeForPointers(tls_begin, tls_end, frontier, "TLS", kReachable); - } else { - // Because LSan should not be loaded with dlopen(), we can assume - // that allocator cache will be part of static TLS image. - CHECK_LE(tls_begin, cache_begin); - CHECK_GE(tls_end, cache_end); - if (tls_begin < cache_begin) - ScanRangeForPointers(tls_begin, cache_begin, frontier, "TLS", - kReachable); - if (tls_end > cache_end) - ScanRangeForPointers(cache_end, tls_end, frontier, "TLS", kReachable); + if (tls_begin) { + LOG_THREADS("TLS at %p-%p.\n", tls_begin, tls_end); + // If the tls and cache ranges don't overlap, scan full tls range, + // otherwise, only scan the non-overlapping portions + if (cache_begin == cache_end || tls_end < cache_begin || + tls_begin > cache_end) { + ScanRangeForPointers(tls_begin, tls_end, frontier, "TLS", kReachable); + } else { + if (tls_begin < cache_begin) + ScanRangeForPointers(tls_begin, cache_begin, frontier, "TLS", + kReachable); + if (tls_end > cache_end) + ScanRangeForPointers(cache_end, tls_end, frontier, "TLS", + kReachable); + } } - if (dtls) { + if (dtls && !DTLSInDestruction(dtls)) { for (uptr j = 0; j < dtls->dtv_size; ++j) { uptr dtls_beg = dtls->dtv[j].beg; uptr dtls_end = dtls_beg + dtls->dtv[j].size; @@ -268,28 +295,36 @@ static void ProcessThreads(SuspendedThreadsList const &suspended_threads, kReachable); } } + } else { + // We are handling a thread with DTLS under destruction. Log about + // this and continue. + LOG_THREADS("Thread %d has DTLS under destruction.\n", os_id); } } } } -static void ProcessRootRegion(Frontier *frontier, uptr root_begin, - uptr root_end) { - MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr begin, end, prot; - while (proc_maps.Next(&begin, &end, - /*offset*/ nullptr, /*filename*/ nullptr, - /*filename_size*/ 0, &prot)) { - uptr intersection_begin = Max(root_begin, begin); - uptr intersection_end = Min(end, root_end); - if (intersection_begin >= intersection_end) continue; - bool is_readable = prot & MemoryMappingLayout::kProtectionRead; - LOG_POINTERS("Root region %p-%p intersects with mapped region %p-%p (%s)\n", - root_begin, root_end, begin, end, - is_readable ? "readable" : "unreadable"); - if (is_readable) - ScanRangeForPointers(intersection_begin, intersection_end, frontier, - "ROOT", kReachable); +void ScanRootRegion(Frontier *frontier, const RootRegion &root_region, + uptr region_begin, uptr region_end, bool is_readable) { + uptr intersection_begin = Max(root_region.begin, region_begin); + uptr intersection_end = Min(region_end, root_region.begin + root_region.size); + if (intersection_begin >= intersection_end) return; + LOG_POINTERS("Root region %p-%p intersects with mapped region %p-%p (%s)\n", + root_region.begin, root_region.begin + root_region.size, + region_begin, region_end, + is_readable ? "readable" : "unreadable"); + if (is_readable) + ScanRangeForPointers(intersection_begin, intersection_end, frontier, "ROOT", + kReachable); +} + +static void ProcessRootRegion(Frontier *frontier, + const RootRegion &root_region) { + MemoryMappingLayout proc_maps(/*cache_enabled*/ true); + MemoryMappedSegment segment; + while (proc_maps.Next(&segment)) { + ScanRootRegion(frontier, root_region, segment.start, segment.end, + segment.IsReadable()); } } @@ -298,9 +333,7 @@ static void ProcessRootRegions(Frontier *frontier) { if (!flags()->use_root_regions) return; CHECK(root_regions); for (uptr i = 0; i < root_regions->size(); i++) { - RootRegion region = (*root_regions)[i]; - uptr begin_addr = reinterpret_cast<uptr>(region.begin); - ProcessRootRegion(frontier, begin_addr, begin_addr + region.size); + ProcessRootRegion(frontier, (*root_regions)[i]); } } @@ -338,6 +371,72 @@ static void CollectIgnoredCb(uptr chunk, void *arg) { } } +static uptr GetCallerPC(u32 stack_id, StackDepotReverseMap *map) { + CHECK(stack_id); + StackTrace stack = map->Get(stack_id); + // The top frame is our malloc/calloc/etc. The next frame is the caller. + if (stack.size >= 2) + return stack.trace[1]; + return 0; +} + +struct InvalidPCParam { + Frontier *frontier; + StackDepotReverseMap *stack_depot_reverse_map; + bool skip_linker_allocations; +}; + +// ForEachChunk callback. If the caller pc is invalid or is within the linker, +// mark as reachable. Called by ProcessPlatformSpecificAllocations. +static void MarkInvalidPCCb(uptr chunk, void *arg) { + CHECK(arg); + InvalidPCParam *param = reinterpret_cast<InvalidPCParam *>(arg); + chunk = GetUserBegin(chunk); + LsanMetadata m(chunk); + if (m.allocated() && m.tag() != kReachable && m.tag() != kIgnored) { + u32 stack_id = m.stack_trace_id(); + uptr caller_pc = 0; + if (stack_id > 0) + caller_pc = GetCallerPC(stack_id, param->stack_depot_reverse_map); + // If caller_pc is unknown, this chunk may be allocated in a coroutine. Mark + // it as reachable, as we can't properly report its allocation stack anyway. + if (caller_pc == 0 || (param->skip_linker_allocations && + GetLinker()->containsAddress(caller_pc))) { + m.set_tag(kReachable); + param->frontier->push_back(chunk); + } + } +} + +// On Linux, handles dynamically allocated TLS blocks by treating all chunks +// allocated from ld-linux.so as reachable. +// Dynamic TLS blocks contain the TLS variables of dynamically loaded modules. +// They are allocated with a __libc_memalign() call in allocate_and_init() +// (elf/dl-tls.c). Glibc won't tell us the address ranges occupied by those +// blocks, but we can make sure they come from our own allocator by intercepting +// __libc_memalign(). On top of that, there is no easy way to reach them. Their +// addresses are stored in a dynamically allocated array (the DTV) which is +// referenced from the static TLS. Unfortunately, we can't just rely on the DTV +// being reachable from the static TLS, and the dynamic TLS being reachable from +// the DTV. This is because the initial DTV is allocated before our interception +// mechanism kicks in, and thus we don't recognize it as allocated memory. We +// can't special-case it either, since we don't know its size. +// Our solution is to include in the root set all allocations made from +// ld-linux.so (which is where allocate_and_init() is implemented). This is +// guaranteed to include all dynamic TLS blocks (and possibly other allocations +// which we don't care about). +// On all other platforms, this simply checks to ensure that the caller pc is +// valid before reporting chunks as leaked. +void ProcessPC(Frontier *frontier) { + StackDepotReverseMap stack_depot_reverse_map; + InvalidPCParam arg; + arg.frontier = frontier; + arg.stack_depot_reverse_map = &stack_depot_reverse_map; + arg.skip_linker_allocations = + flags()->use_tls && flags()->use_ld_allocations && GetLinker() != nullptr; + ForEachChunk(MarkInvalidPCCb, &arg); +} + // Sets the appropriate tag on each chunk. static void ClassifyAllChunks(SuspendedThreadsList const &suspended_threads) { // Holds the flood fill frontier. @@ -349,11 +448,13 @@ static void ClassifyAllChunks(SuspendedThreadsList const &suspended_threads) { ProcessRootRegions(&frontier); FloodFillTag(&frontier, kReachable); + CHECK_EQ(0, frontier.size()); + ProcessPC(&frontier); + // The check here is relatively expensive, so we do this in a separate flood // fill. That way we can skip the check for chunks that are reachable // otherwise. LOG_POINTERS("Processing platform-specific allocations.\n"); - CHECK_EQ(0, frontier.size()); ProcessPlatformSpecificAllocations(&frontier); FloodFillTag(&frontier, kReachable); @@ -475,18 +576,16 @@ static bool CheckForLeaks() { return false; } +static bool has_reported_leaks = false; +bool HasReportedLeaks() { return has_reported_leaks; } + void DoLeakCheck() { BlockingMutexLock l(&global_mutex); static bool already_done; if (already_done) return; already_done = true; - bool have_leaks = CheckForLeaks(); - if (!have_leaks) { - return; - } - if (common_flags()->exitcode) { - Die(); - } + has_reported_leaks = CheckForLeaks(); + if (has_reported_leaks) HandleLeaks(); } static int DoRecoverableLeakCheck() { @@ -689,7 +788,7 @@ void __lsan_register_root_region(const void *begin, uptr size) { #if CAN_SANITIZE_LEAKS BlockingMutexLock l(&global_mutex); CHECK(root_regions); - RootRegion region = {begin, size}; + RootRegion region = {reinterpret_cast<uptr>(begin), size}; root_regions->push_back(region); VReport(1, "Registered root region at %p of size %llu\n", begin, size); #endif // CAN_SANITIZE_LEAKS @@ -703,7 +802,7 @@ void __lsan_unregister_root_region(const void *begin, uptr size) { bool removed = false; for (uptr i = 0; i < root_regions->size(); i++) { RootRegion region = (*root_regions)[i]; - if (region.begin == begin && region.size == size) { + if (region.begin == reinterpret_cast<uptr>(begin) && region.size == size) { removed = true; uptr last_index = root_regions->size() - 1; (*root_regions)[i] = (*root_regions)[last_index]; diff --git a/contrib/compiler-rt/lib/lsan/lsan_common.h b/contrib/compiler-rt/lib/lsan/lsan_common.h index 890ce65..31bf3eb 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_common.h +++ b/contrib/compiler-rt/lib/lsan/lsan_common.h @@ -22,8 +22,24 @@ #include "sanitizer_common/sanitizer_stoptheworld.h" #include "sanitizer_common/sanitizer_symbolizer.h" -#if (SANITIZER_LINUX && !SANITIZER_ANDROID) && (SANITIZER_WORDSIZE == 64) \ - && (defined(__x86_64__) || defined(__mips64) || defined(__aarch64__)) +// LeakSanitizer relies on some Glibc's internals (e.g. TLS machinery) thus +// supported for Linux only. Also, LSan doesn't like 32 bit architectures +// because of "small" (4 bytes) pointer size that leads to high false negative +// ratio on large leaks. But we still want to have it for some 32 bit arches +// (e.g. x86), see https://github.com/google/sanitizers/issues/403. +// To enable LeakSanitizer on new architecture, one need to implement +// internal_clone function as well as (probably) adjust TLS machinery for +// new architecture inside sanitizer library. +#if (SANITIZER_LINUX && !SANITIZER_ANDROID || SANITIZER_MAC) && \ + (SANITIZER_WORDSIZE == 64) && \ + (defined(__x86_64__) || defined(__mips64) || defined(__aarch64__) || \ + defined(__powerpc64__)) +#define CAN_SANITIZE_LEAKS 1 +#elif defined(__i386__) && \ + (SANITIZER_LINUX && !SANITIZER_ANDROID || SANITIZER_MAC) +#define CAN_SANITIZE_LEAKS 1 +#elif defined(__arm__) && \ + SANITIZER_LINUX && !SANITIZER_ANDROID #define CAN_SANITIZE_LEAKS 1 #else #define CAN_SANITIZE_LEAKS 0 @@ -44,6 +60,8 @@ enum ChunkTag { kIgnored = 3 }; +const u32 kInvalidTid = (u32) -1; + struct Flags { #define LSAN_FLAG(Type, Name, DefaultValue, Description) Type Name; #include "lsan_flags.inc" @@ -101,12 +119,22 @@ typedef InternalMmapVector<uptr> Frontier; void InitializePlatformSpecificModules(); void ProcessGlobalRegions(Frontier *frontier); void ProcessPlatformSpecificAllocations(Frontier *frontier); + +struct RootRegion { + uptr begin; + uptr size; +}; + +InternalMmapVector<RootRegion> const *GetRootRegions(); +void ScanRootRegion(Frontier *frontier, RootRegion const ®ion, + uptr region_begin, uptr region_end, bool is_readable); // Run stoptheworld while holding any platform-specific locks. void DoStopTheWorld(StopTheWorldCallback callback, void* argument); void ScanRangeForPointers(uptr begin, uptr end, Frontier *frontier, const char *region_type, ChunkTag tag); +void ScanGlobalRange(uptr begin, uptr end, Frontier *frontier); enum IgnoreObjectResult { kIgnoreObjectSuccess, @@ -117,6 +145,7 @@ enum IgnoreObjectResult { // Functions called from the parent tool. void InitCommonLsan(); void DoLeakCheck(); +void DisableCounterUnderflow(); bool DisabledInThisThread(); // Used to implement __lsan::ScopedDisabler. @@ -129,13 +158,36 @@ struct ScopedInterceptorDisabler { ~ScopedInterceptorDisabler() { EnableInThisThread(); } }; +// According to Itanium C++ ABI array cookie is a one word containing +// size of allocated array. +static inline bool IsItaniumABIArrayCookie(uptr chunk_beg, uptr chunk_size, + uptr addr) { + return chunk_size == sizeof(uptr) && chunk_beg + chunk_size == addr && + *reinterpret_cast<uptr *>(chunk_beg) == 0; +} + +// According to ARM C++ ABI array cookie consists of two words: +// struct array_cookie { +// std::size_t element_size; // element_size != 0 +// std::size_t element_count; +// }; +static inline bool IsARMABIArrayCookie(uptr chunk_beg, uptr chunk_size, + uptr addr) { + return chunk_size == 2 * sizeof(uptr) && chunk_beg + chunk_size == addr && + *reinterpret_cast<uptr *>(chunk_beg + sizeof(uptr)) == 0; +} + // Special case for "new T[0]" where T is a type with DTOR. -// new T[0] will allocate one word for the array size (0) and store a pointer -// to the end of allocated chunk. +// new T[0] will allocate a cookie (one or two words) for the array size (0) +// and store a pointer to the end of allocated chunk. The actual cookie layout +// varies between platforms according to their C++ ABI implementation. inline bool IsSpecialCaseOfOperatorNew0(uptr chunk_beg, uptr chunk_size, uptr addr) { - return chunk_size == sizeof(uptr) && chunk_beg + chunk_size == addr && - *reinterpret_cast<uptr *>(chunk_beg) == 0; +#if defined(__arm__) + return IsARMABIArrayCookie(chunk_beg, chunk_size, addr); +#else + return IsItaniumABIArrayCookie(chunk_beg, chunk_size, addr); +#endif } // The following must be implemented in the parent tool. @@ -151,10 +203,10 @@ bool WordIsPoisoned(uptr addr); // Wrappers for ThreadRegistry access. void LockThreadRegistry(); void UnlockThreadRegistry(); -bool GetThreadRangesLocked(uptr os_id, uptr *stack_begin, uptr *stack_end, +bool GetThreadRangesLocked(tid_t os_id, uptr *stack_begin, uptr *stack_end, uptr *tls_begin, uptr *tls_end, uptr *cache_begin, uptr *cache_end, DTLS **dtls); -void ForEachExtraStackRange(uptr os_id, RangeIteratorCallback callback, +void ForEachExtraStackRange(tid_t os_id, RangeIteratorCallback callback, void *arg); // If called from the main thread, updates the main thread's TID in the thread // registry. We need this to handle processes that fork() without a subsequent @@ -170,6 +222,16 @@ uptr PointsIntoChunk(void *p); uptr GetUserBegin(uptr chunk); // Helper for __lsan_ignore_object(). IgnoreObjectResult IgnoreObjectLocked(const void *p); + +// Return the linker module, if valid for the platform. +LoadedModule *GetLinker(); + +// Return true if LSan has finished leak checking and reported leaks. +bool HasReportedLeaks(); + +// Run platform-specific leak handlers. +void HandleLeaks(); + // Wrapper for chunk metadata operations. class LsanMetadata { public: diff --git a/contrib/compiler-rt/lib/lsan/lsan_common_linux.cc b/contrib/compiler-rt/lib/lsan/lsan_common_linux.cc index f6154d8..5042c7b 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_common_linux.cc +++ b/contrib/compiler-rt/lib/lsan/lsan_common_linux.cc @@ -34,6 +34,17 @@ static bool IsLinker(const char* full_name) { return LibraryNameIs(full_name, kLinkerName); } +__attribute__((tls_model("initial-exec"))) +THREADLOCAL int disable_counter; +bool DisabledInThisThread() { return disable_counter > 0; } +void DisableInThisThread() { disable_counter++; } +void EnableInThisThread() { + if (disable_counter == 0) { + DisableCounterUnderflow(); + } + disable_counter--; +} + void InitializePlatformSpecificModules() { ListOfModules modules; modules.init(); @@ -51,8 +62,10 @@ void InitializePlatformSpecificModules() { return; } } - VReport(1, "LeakSanitizer: Dynamic linker not found. " - "TLS will not be handled correctly.\n"); + if (linker == nullptr) { + VReport(1, "LeakSanitizer: Dynamic linker not found. " + "TLS will not be handled correctly.\n"); + } } static int ProcessGlobalRegionsCallback(struct dl_phdr_info *info, size_t size, @@ -67,20 +80,7 @@ static int ProcessGlobalRegionsCallback(struct dl_phdr_info *info, size_t size, continue; uptr begin = info->dlpi_addr + phdr->p_vaddr; uptr end = begin + phdr->p_memsz; - uptr allocator_begin = 0, allocator_end = 0; - GetAllocatorGlobalRange(&allocator_begin, &allocator_end); - if (begin <= allocator_begin && allocator_begin < end) { - CHECK_LE(allocator_begin, allocator_end); - CHECK_LE(allocator_end, end); - if (begin < allocator_begin) - ScanRangeForPointers(begin, allocator_begin, frontier, "GLOBAL", - kReachable); - if (allocator_end < end) - ScanRangeForPointers(allocator_end, end, frontier, "GLOBAL", - kReachable); - } else { - ScanRangeForPointers(begin, end, frontier, "GLOBAL", kReachable); - } + ScanGlobalRange(begin, end, frontier); } return 0; } @@ -91,76 +91,22 @@ void ProcessGlobalRegions(Frontier *frontier) { dl_iterate_phdr(ProcessGlobalRegionsCallback, frontier); } -static uptr GetCallerPC(u32 stack_id, StackDepotReverseMap *map) { - CHECK(stack_id); - StackTrace stack = map->Get(stack_id); - // The top frame is our malloc/calloc/etc. The next frame is the caller. - if (stack.size >= 2) - return stack.trace[1]; - return 0; -} +LoadedModule *GetLinker() { return linker; } -struct ProcessPlatformAllocParam { - Frontier *frontier; - StackDepotReverseMap *stack_depot_reverse_map; - bool skip_linker_allocations; -}; - -// ForEachChunk callback. Identifies unreachable chunks which must be treated as -// reachable. Marks them as reachable and adds them to the frontier. -static void ProcessPlatformSpecificAllocationsCb(uptr chunk, void *arg) { - CHECK(arg); - ProcessPlatformAllocParam *param = - reinterpret_cast<ProcessPlatformAllocParam *>(arg); - chunk = GetUserBegin(chunk); - LsanMetadata m(chunk); - if (m.allocated() && m.tag() != kReachable && m.tag() != kIgnored) { - u32 stack_id = m.stack_trace_id(); - uptr caller_pc = 0; - if (stack_id > 0) - caller_pc = GetCallerPC(stack_id, param->stack_depot_reverse_map); - // If caller_pc is unknown, this chunk may be allocated in a coroutine. Mark - // it as reachable, as we can't properly report its allocation stack anyway. - if (caller_pc == 0 || (param->skip_linker_allocations && - linker->containsAddress(caller_pc))) { - m.set_tag(kReachable); - param->frontier->push_back(chunk); - } - } -} - -// Handles dynamically allocated TLS blocks by treating all chunks allocated -// from ld-linux.so as reachable. -// Dynamic TLS blocks contain the TLS variables of dynamically loaded modules. -// They are allocated with a __libc_memalign() call in allocate_and_init() -// (elf/dl-tls.c). Glibc won't tell us the address ranges occupied by those -// blocks, but we can make sure they come from our own allocator by intercepting -// __libc_memalign(). On top of that, there is no easy way to reach them. Their -// addresses are stored in a dynamically allocated array (the DTV) which is -// referenced from the static TLS. Unfortunately, we can't just rely on the DTV -// being reachable from the static TLS, and the dynamic TLS being reachable from -// the DTV. This is because the initial DTV is allocated before our interception -// mechanism kicks in, and thus we don't recognize it as allocated memory. We -// can't special-case it either, since we don't know its size. -// Our solution is to include in the root set all allocations made from -// ld-linux.so (which is where allocate_and_init() is implemented). This is -// guaranteed to include all dynamic TLS blocks (and possibly other allocations -// which we don't care about). -void ProcessPlatformSpecificAllocations(Frontier *frontier) { - StackDepotReverseMap stack_depot_reverse_map; - ProcessPlatformAllocParam arg; - arg.frontier = frontier; - arg.stack_depot_reverse_map = &stack_depot_reverse_map; - arg.skip_linker_allocations = - flags()->use_tls && flags()->use_ld_allocations && linker != nullptr; - ForEachChunk(ProcessPlatformSpecificAllocationsCb, &arg); -} +void ProcessPlatformSpecificAllocations(Frontier *frontier) {} struct DoStopTheWorldParam { StopTheWorldCallback callback; void *argument; }; +// While calling Die() here is undefined behavior and can potentially +// cause race conditions, it isn't possible to intercept exit on linux, +// so we have no choice but to call Die() from the atexit handler. +void HandleLeaks() { + if (common_flags()->exitcode) Die(); +} + static int DoStopTheWorldCallback(struct dl_phdr_info *info, size_t size, void *data) { DoStopTheWorldParam *param = reinterpret_cast<DoStopTheWorldParam *>(data); diff --git a/contrib/compiler-rt/lib/lsan/lsan_common_mac.cc b/contrib/compiler-rt/lib/lsan/lsan_common_mac.cc new file mode 100644 index 0000000..ade9434 --- /dev/null +++ b/contrib/compiler-rt/lib/lsan/lsan_common_mac.cc @@ -0,0 +1,178 @@ +//=-- lsan_common_mac.cc --------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of LeakSanitizer. +// Implementation of common leak checking functionality. Darwin-specific code. +// +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" +#include "lsan_common.h" + +#if CAN_SANITIZE_LEAKS && SANITIZER_MAC + +#include "sanitizer_common/sanitizer_allocator_internal.h" +#include "lsan_allocator.h" + +#include <pthread.h> + +#include <mach/mach.h> + +namespace __lsan { + +typedef struct { + int disable_counter; + u32 current_thread_id; + AllocatorCache cache; +} thread_local_data_t; + +static pthread_key_t key; +static pthread_once_t key_once = PTHREAD_ONCE_INIT; + +// The main thread destructor requires the current thread id, +// so we can't destroy it until it's been used and reset to invalid tid +void restore_tid_data(void *ptr) { + thread_local_data_t *data = (thread_local_data_t *)ptr; + if (data->current_thread_id != kInvalidTid) + pthread_setspecific(key, data); +} + +static void make_tls_key() { + CHECK_EQ(pthread_key_create(&key, restore_tid_data), 0); +} + +static thread_local_data_t *get_tls_val(bool alloc) { + pthread_once(&key_once, make_tls_key); + + thread_local_data_t *ptr = (thread_local_data_t *)pthread_getspecific(key); + if (ptr == NULL && alloc) { + ptr = (thread_local_data_t *)InternalAlloc(sizeof(*ptr)); + ptr->disable_counter = 0; + ptr->current_thread_id = kInvalidTid; + ptr->cache = AllocatorCache(); + pthread_setspecific(key, ptr); + } + + return ptr; +} + +bool DisabledInThisThread() { + thread_local_data_t *data = get_tls_val(false); + return data ? data->disable_counter > 0 : false; +} + +void DisableInThisThread() { ++get_tls_val(true)->disable_counter; } + +void EnableInThisThread() { + int *disable_counter = &get_tls_val(true)->disable_counter; + if (*disable_counter == 0) { + DisableCounterUnderflow(); + } + --*disable_counter; +} + +u32 GetCurrentThread() { + thread_local_data_t *data = get_tls_val(false); + return data ? data->current_thread_id : kInvalidTid; +} + +void SetCurrentThread(u32 tid) { get_tls_val(true)->current_thread_id = tid; } + +AllocatorCache *GetAllocatorCache() { return &get_tls_val(true)->cache; } + +LoadedModule *GetLinker() { return nullptr; } + +// Required on Linux for initialization of TLS behavior, but should not be +// required on Darwin. +void InitializePlatformSpecificModules() {} + +// Scans global variables for heap pointers. +void ProcessGlobalRegions(Frontier *frontier) { + MemoryMappingLayout memory_mapping(false); + InternalMmapVector<LoadedModule> modules(/*initial_capacity*/ 128); + memory_mapping.DumpListOfModules(&modules); + for (uptr i = 0; i < modules.size(); ++i) { + // Even when global scanning is disabled, we still need to scan + // system libraries for stashed pointers + if (!flags()->use_globals && modules[i].instrumented()) continue; + + for (const __sanitizer::LoadedModule::AddressRange &range : + modules[i].ranges()) { + // Sections storing global variables are writable and non-executable + if (range.executable || !range.writable) continue; + + ScanGlobalRange(range.beg, range.end, frontier); + } + } +} + +void ProcessPlatformSpecificAllocations(Frontier *frontier) { + mach_port_name_t port; + if (task_for_pid(mach_task_self(), internal_getpid(), &port) + != KERN_SUCCESS) { + return; + } + + unsigned depth = 1; + vm_size_t size = 0; + vm_address_t address = 0; + kern_return_t err = KERN_SUCCESS; + mach_msg_type_number_t count = VM_REGION_SUBMAP_INFO_COUNT_64; + + InternalMmapVector<RootRegion> const *root_regions = GetRootRegions(); + + while (err == KERN_SUCCESS) { + struct vm_region_submap_info_64 info; + err = vm_region_recurse_64(port, &address, &size, &depth, + (vm_region_info_t)&info, &count); + + uptr end_address = address + size; + + // libxpc stashes some pointers in the Kernel Alloc Once page, + // make sure not to report those as leaks. + if (info.user_tag == VM_MEMORY_OS_ALLOC_ONCE) { + ScanRangeForPointers(address, end_address, frontier, "GLOBAL", + kReachable); + + // Recursing over the full memory map is very slow, break out + // early if we don't need the full iteration. + if (!flags()->use_root_regions || !root_regions->size()) + break; + } + + // This additional root region scan is required on Darwin in order to + // detect root regions contained within mmap'd memory regions, because + // the Darwin implementation of sanitizer_procmaps traverses images + // as loaded by dyld, and not the complete set of all memory regions. + // + // TODO(fjricci) - remove this once sanitizer_procmaps_mac has the same + // behavior as sanitizer_procmaps_linux and traverses all memory regions + if (flags()->use_root_regions) { + for (uptr i = 0; i < root_regions->size(); i++) { + ScanRootRegion(frontier, (*root_regions)[i], address, end_address, + info.protection & kProtectionRead); + } + } + + address = end_address; + } +} + +// On darwin, we can intercept _exit gracefully, and return a failing exit code +// if required at that point. Calling Die() here is undefined behavior and +// causes rare race conditions. +void HandleLeaks() {} + +void DoStopTheWorld(StopTheWorldCallback callback, void *argument) { + StopTheWorld(callback, argument); +} + +} // namespace __lsan + +#endif // CAN_SANITIZE_LEAKS && SANITIZER_MAC diff --git a/contrib/compiler-rt/lib/lsan/lsan_interceptors.cc b/contrib/compiler-rt/lib/lsan/lsan_interceptors.cc index 1219017..168868b 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_interceptors.cc +++ b/contrib/compiler-rt/lib/lsan/lsan_interceptors.cc @@ -21,12 +21,15 @@ #include "sanitizer_common/sanitizer_linux.h" #include "sanitizer_common/sanitizer_platform_interceptors.h" #include "sanitizer_common/sanitizer_platform_limits_posix.h" +#include "sanitizer_common/sanitizer_posix.h" #include "sanitizer_common/sanitizer_tls_get_addr.h" #include "lsan.h" #include "lsan_allocator.h" #include "lsan_common.h" #include "lsan_thread.h" +#include <stddef.h> + using namespace __lsan; extern "C" { @@ -37,29 +40,22 @@ int pthread_key_create(unsigned *key, void (*destructor)(void* v)); int pthread_setspecific(unsigned key, const void *v); } -#define ENSURE_LSAN_INITED do { \ - CHECK(!lsan_init_is_running); \ - if (!lsan_inited) \ - __lsan_init(); \ -} while (0) - ///// Malloc/free interceptors. ///// -const bool kAlwaysClearMemory = true; - namespace std { struct nothrow_t; } +#if !SANITIZER_MAC INTERCEPTOR(void*, malloc, uptr size) { ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - return Allocate(stack, size, 1, kAlwaysClearMemory); + return lsan_malloc(size, stack); } INTERCEPTOR(void, free, void *p) { ENSURE_LSAN_INITED; - Deallocate(p); + lsan_free(p); } INTERCEPTOR(void*, calloc, uptr nmemb, uptr size) { @@ -74,31 +70,44 @@ INTERCEPTOR(void*, calloc, uptr nmemb, uptr size) { CHECK(allocated < kCallocPoolSize); return mem; } - if (CallocShouldReturnNullDueToOverflow(size, nmemb)) return nullptr; ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - size *= nmemb; - return Allocate(stack, size, 1, true); + return lsan_calloc(nmemb, size, stack); } INTERCEPTOR(void*, realloc, void *q, uptr size) { ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - return Reallocate(stack, q, size, 1); + return lsan_realloc(q, size, stack); +} + +INTERCEPTOR(int, posix_memalign, void **memptr, uptr alignment, uptr size) { + ENSURE_LSAN_INITED; + GET_STACK_TRACE_MALLOC; + *memptr = lsan_memalign(alignment, size, stack); + // FIXME: Return ENOMEM if user requested more than max alloc size. + return 0; +} + +INTERCEPTOR(void*, valloc, uptr size) { + ENSURE_LSAN_INITED; + GET_STACK_TRACE_MALLOC; + return lsan_valloc(size, stack); } +#endif #if SANITIZER_INTERCEPT_MEMALIGN INTERCEPTOR(void*, memalign, uptr alignment, uptr size) { ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - return Allocate(stack, size, alignment, kAlwaysClearMemory); + return lsan_memalign(alignment, size, stack); } #define LSAN_MAYBE_INTERCEPT_MEMALIGN INTERCEPT_FUNCTION(memalign) INTERCEPTOR(void *, __libc_memalign, uptr alignment, uptr size) { ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - void *res = Allocate(stack, size, alignment, kAlwaysClearMemory); + void *res = lsan_memalign(alignment, size, stack); DTLS_on_libc_memalign(res, size); return res; } @@ -108,32 +117,27 @@ INTERCEPTOR(void *, __libc_memalign, uptr alignment, uptr size) { #define LSAN_MAYBE_INTERCEPT___LIBC_MEMALIGN #endif // SANITIZER_INTERCEPT_MEMALIGN +#if SANITIZER_INTERCEPT_ALIGNED_ALLOC INTERCEPTOR(void*, aligned_alloc, uptr alignment, uptr size) { ENSURE_LSAN_INITED; GET_STACK_TRACE_MALLOC; - return Allocate(stack, size, alignment, kAlwaysClearMemory); -} - -INTERCEPTOR(int, posix_memalign, void **memptr, uptr alignment, uptr size) { - ENSURE_LSAN_INITED; - GET_STACK_TRACE_MALLOC; - *memptr = Allocate(stack, size, alignment, kAlwaysClearMemory); - // FIXME: Return ENOMEM if user requested more than max alloc size. - return 0; -} - -INTERCEPTOR(void*, valloc, uptr size) { - ENSURE_LSAN_INITED; - GET_STACK_TRACE_MALLOC; - if (size == 0) - size = GetPageSizeCached(); - return Allocate(stack, size, GetPageSizeCached(), kAlwaysClearMemory); + return lsan_memalign(alignment, size, stack); } +#define LSAN_MAYBE_INTERCEPT_ALIGNED_ALLOC INTERCEPT_FUNCTION(aligned_alloc) +#else +#define LSAN_MAYBE_INTERCEPT_ALIGNED_ALLOC +#endif +#if SANITIZER_INTERCEPT_MALLOC_USABLE_SIZE INTERCEPTOR(uptr, malloc_usable_size, void *ptr) { ENSURE_LSAN_INITED; return GetMallocUsableSize(ptr); } +#define LSAN_MAYBE_INTERCEPT_MALLOC_USABLE_SIZE \ + INTERCEPT_FUNCTION(malloc_usable_size) +#else +#define LSAN_MAYBE_INTERCEPT_MALLOC_USABLE_SIZE +#endif #if SANITIZER_INTERCEPT_MALLOPT_AND_MALLINFO struct fake_mallinfo { @@ -180,24 +184,52 @@ INTERCEPTOR(void, cfree, void *p) ALIAS(WRAPPER_NAME(free)); #define LSAN_MAYBE_INTERCEPT_CFREE #endif // SANITIZER_INTERCEPT_CFREE -#define OPERATOR_NEW_BODY \ - ENSURE_LSAN_INITED; \ - GET_STACK_TRACE_MALLOC; \ - return Allocate(stack, size, 1, kAlwaysClearMemory); +#if SANITIZER_INTERCEPT_MCHECK_MPROBE +INTERCEPTOR(int, mcheck, void (*abortfunc)(int mstatus)) { + return 0; +} -INTERCEPTOR_ATTRIBUTE -void *operator new(uptr size) { OPERATOR_NEW_BODY; } -INTERCEPTOR_ATTRIBUTE -void *operator new[](uptr size) { OPERATOR_NEW_BODY; } -INTERCEPTOR_ATTRIBUTE -void *operator new(uptr size, std::nothrow_t const&) { OPERATOR_NEW_BODY; } -INTERCEPTOR_ATTRIBUTE -void *operator new[](uptr size, std::nothrow_t const&) { OPERATOR_NEW_BODY; } +INTERCEPTOR(int, mcheck_pedantic, void (*abortfunc)(int mstatus)) { + return 0; +} + +INTERCEPTOR(int, mprobe, void *ptr) { + return 0; +} +#endif // SANITIZER_INTERCEPT_MCHECK_MPROBE + + +// TODO(alekseys): throw std::bad_alloc instead of dying on OOM. +#define OPERATOR_NEW_BODY(nothrow) \ + ENSURE_LSAN_INITED; \ + GET_STACK_TRACE_MALLOC; \ + void *res = Allocate(stack, size, 1, kAlwaysClearMemory);\ + if (!nothrow && UNLIKELY(!res)) DieOnFailure::OnOOM();\ + return res; #define OPERATOR_DELETE_BODY \ ENSURE_LSAN_INITED; \ Deallocate(ptr); +// On OS X it's not enough to just provide our own 'operator new' and +// 'operator delete' implementations, because they're going to be in the runtime +// dylib, and the main executable will depend on both the runtime dylib and +// libstdc++, each of has its implementation of new and delete. +// To make sure that C++ allocation/deallocation operators are overridden on +// OS X we need to intercept them using their mangled names. +#if !SANITIZER_MAC + +INTERCEPTOR_ATTRIBUTE +void *operator new(size_t size) { OPERATOR_NEW_BODY(false /*nothrow*/); } +INTERCEPTOR_ATTRIBUTE +void *operator new[](size_t size) { OPERATOR_NEW_BODY(false /*nothrow*/); } +INTERCEPTOR_ATTRIBUTE +void *operator new(size_t size, std::nothrow_t const&) +{ OPERATOR_NEW_BODY(true /*nothrow*/); } +INTERCEPTOR_ATTRIBUTE +void *operator new[](size_t size, std::nothrow_t const&) +{ OPERATOR_NEW_BODY(true /*nothrow*/); } + INTERCEPTOR_ATTRIBUTE void operator delete(void *ptr) NOEXCEPT { OPERATOR_DELETE_BODY; } INTERCEPTOR_ATTRIBUTE @@ -205,9 +237,31 @@ void operator delete[](void *ptr) NOEXCEPT { OPERATOR_DELETE_BODY; } INTERCEPTOR_ATTRIBUTE void operator delete(void *ptr, std::nothrow_t const&) { OPERATOR_DELETE_BODY; } INTERCEPTOR_ATTRIBUTE -void operator delete[](void *ptr, std::nothrow_t const &) { - OPERATOR_DELETE_BODY; -} +void operator delete[](void *ptr, std::nothrow_t const &) +{ OPERATOR_DELETE_BODY; } + +#else // SANITIZER_MAC + +INTERCEPTOR(void *, _Znwm, size_t size) +{ OPERATOR_NEW_BODY(false /*nothrow*/); } +INTERCEPTOR(void *, _Znam, size_t size) +{ OPERATOR_NEW_BODY(false /*nothrow*/); } +INTERCEPTOR(void *, _ZnwmRKSt9nothrow_t, size_t size, std::nothrow_t const&) +{ OPERATOR_NEW_BODY(true /*nothrow*/); } +INTERCEPTOR(void *, _ZnamRKSt9nothrow_t, size_t size, std::nothrow_t const&) +{ OPERATOR_NEW_BODY(true /*nothrow*/); } + +INTERCEPTOR(void, _ZdlPv, void *ptr) +{ OPERATOR_DELETE_BODY; } +INTERCEPTOR(void, _ZdaPv, void *ptr) +{ OPERATOR_DELETE_BODY; } +INTERCEPTOR(void, _ZdlPvRKSt9nothrow_t, void *ptr, std::nothrow_t const&) +{ OPERATOR_DELETE_BODY; } +INTERCEPTOR(void, _ZdaPvRKSt9nothrow_t, void *ptr, std::nothrow_t const&) +{ OPERATOR_DELETE_BODY; } + +#endif // !SANITIZER_MAC + ///// Thread initialization and finalization. ///// @@ -277,7 +331,8 @@ INTERCEPTOR(int, pthread_create, void *th, void *attr, res = REAL(pthread_create)(th, attr, __lsan_thread_start_func, &p); } if (res == 0) { - int tid = ThreadCreate(GetCurrentThread(), *(uptr *)th, detached); + int tid = ThreadCreate(GetCurrentThread(), *(uptr *)th, + IsStateDetached(detached)); CHECK_NE(tid, 0); atomic_store(&p.tid, tid, memory_order_release); while (atomic_load(&p.tid, memory_order_acquire) != 0) @@ -297,6 +352,11 @@ INTERCEPTOR(int, pthread_join, void *th, void **ret) { return res; } +INTERCEPTOR(void, _exit, int status) { + if (status == 0 && HasReportedLeaks()) status = common_flags()->exitcode; + REAL(_exit)(status); +} + namespace __lsan { void InitializeInterceptors() { @@ -307,15 +367,16 @@ void InitializeInterceptors() { INTERCEPT_FUNCTION(realloc); LSAN_MAYBE_INTERCEPT_MEMALIGN; LSAN_MAYBE_INTERCEPT___LIBC_MEMALIGN; - INTERCEPT_FUNCTION(aligned_alloc); + LSAN_MAYBE_INTERCEPT_ALIGNED_ALLOC; INTERCEPT_FUNCTION(posix_memalign); INTERCEPT_FUNCTION(valloc); LSAN_MAYBE_INTERCEPT_PVALLOC; - INTERCEPT_FUNCTION(malloc_usable_size); + LSAN_MAYBE_INTERCEPT_MALLOC_USABLE_SIZE; LSAN_MAYBE_INTERCEPT_MALLINFO; LSAN_MAYBE_INTERCEPT_MALLOPT; INTERCEPT_FUNCTION(pthread_create); INTERCEPT_FUNCTION(pthread_join); + INTERCEPT_FUNCTION(_exit); if (pthread_key_create(&g_thread_finalize_key, &thread_finalize)) { Report("LeakSanitizer: failed to create thread key.\n"); diff --git a/contrib/compiler-rt/lib/lsan/lsan_linux.cc b/contrib/compiler-rt/lib/lsan/lsan_linux.cc new file mode 100644 index 0000000..c9749c7 --- /dev/null +++ b/contrib/compiler-rt/lib/lsan/lsan_linux.cc @@ -0,0 +1,33 @@ +//=-- lsan_linux.cc -------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of LeakSanitizer. Linux-specific code. +// +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" + +#if SANITIZER_LINUX + +#include "lsan_allocator.h" + +namespace __lsan { + +static THREADLOCAL u32 current_thread_tid = kInvalidTid; +u32 GetCurrentThread() { return current_thread_tid; } +void SetCurrentThread(u32 tid) { current_thread_tid = tid; } + +static THREADLOCAL AllocatorCache allocator_cache; +AllocatorCache *GetAllocatorCache() { return &allocator_cache; } + +void ReplaceSystemMalloc() {} + +} // namespace __lsan + +#endif // SANITIZER_LINUX diff --git a/contrib/compiler-rt/lib/lsan/lsan_mac.cc b/contrib/compiler-rt/lib/lsan/lsan_mac.cc new file mode 100644 index 0000000..1a6f5f4 --- /dev/null +++ b/contrib/compiler-rt/lib/lsan/lsan_mac.cc @@ -0,0 +1,192 @@ +//===-- lsan_mac.cc -------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of LeakSanitizer, a memory leak checker. +// +// Mac-specific details. +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" +#if SANITIZER_MAC + +#include "interception/interception.h" +#include "lsan.h" +#include "lsan_allocator.h" +#include "lsan_thread.h" + +#include <pthread.h> + +namespace __lsan { +// Support for the following functions from libdispatch on Mac OS: +// dispatch_async_f() +// dispatch_async() +// dispatch_sync_f() +// dispatch_sync() +// dispatch_after_f() +// dispatch_after() +// dispatch_group_async_f() +// dispatch_group_async() +// TODO(glider): libdispatch API contains other functions that we don't support +// yet. +// +// dispatch_sync() and dispatch_sync_f() are synchronous, although chances are +// they can cause jobs to run on a thread different from the current one. +// TODO(glider): if so, we need a test for this (otherwise we should remove +// them). +// +// The following functions use dispatch_barrier_async_f() (which isn't a library +// function but is exported) and are thus supported: +// dispatch_source_set_cancel_handler_f() +// dispatch_source_set_cancel_handler() +// dispatch_source_set_event_handler_f() +// dispatch_source_set_event_handler() +// +// The reference manual for Grand Central Dispatch is available at +// http://developer.apple.com/library/mac/#documentation/Performance/Reference/GCD_libdispatch_Ref/Reference/reference.html +// The implementation details are at +// http://libdispatch.macosforge.org/trac/browser/trunk/src/queue.c + +typedef void *dispatch_group_t; +typedef void *dispatch_queue_t; +typedef void *dispatch_source_t; +typedef u64 dispatch_time_t; +typedef void (*dispatch_function_t)(void *block); +typedef void *(*worker_t)(void *block); + +// A wrapper for the ObjC blocks used to support libdispatch. +typedef struct { + void *block; + dispatch_function_t func; + u32 parent_tid; +} lsan_block_context_t; + +ALWAYS_INLINE +void lsan_register_worker_thread(int parent_tid) { + if (GetCurrentThread() == kInvalidTid) { + u32 tid = ThreadCreate(parent_tid, 0, true); + ThreadStart(tid, GetTid()); + SetCurrentThread(tid); + } +} + +// For use by only those functions that allocated the context via +// alloc_lsan_context(). +extern "C" void lsan_dispatch_call_block_and_release(void *block) { + lsan_block_context_t *context = (lsan_block_context_t *)block; + VReport(2, + "lsan_dispatch_call_block_and_release(): " + "context: %p, pthread_self: %p\n", + block, pthread_self()); + lsan_register_worker_thread(context->parent_tid); + // Call the original dispatcher for the block. + context->func(context->block); + lsan_free(context); +} + +} // namespace __lsan + +using namespace __lsan; // NOLINT + +// Wrap |ctxt| and |func| into an lsan_block_context_t. +// The caller retains control of the allocated context. +extern "C" lsan_block_context_t *alloc_lsan_context(void *ctxt, + dispatch_function_t func) { + GET_STACK_TRACE_THREAD; + lsan_block_context_t *lsan_ctxt = + (lsan_block_context_t *)lsan_malloc(sizeof(lsan_block_context_t), stack); + lsan_ctxt->block = ctxt; + lsan_ctxt->func = func; + lsan_ctxt->parent_tid = GetCurrentThread(); + return lsan_ctxt; +} + +// Define interceptor for dispatch_*_f function with the three most common +// parameters: dispatch_queue_t, context, dispatch_function_t. +#define INTERCEPT_DISPATCH_X_F_3(dispatch_x_f) \ + INTERCEPTOR(void, dispatch_x_f, dispatch_queue_t dq, void *ctxt, \ + dispatch_function_t func) { \ + lsan_block_context_t *lsan_ctxt = alloc_lsan_context(ctxt, func); \ + return REAL(dispatch_x_f)(dq, (void *)lsan_ctxt, \ + lsan_dispatch_call_block_and_release); \ + } + +INTERCEPT_DISPATCH_X_F_3(dispatch_async_f) +INTERCEPT_DISPATCH_X_F_3(dispatch_sync_f) +INTERCEPT_DISPATCH_X_F_3(dispatch_barrier_async_f) + +INTERCEPTOR(void, dispatch_after_f, dispatch_time_t when, dispatch_queue_t dq, + void *ctxt, dispatch_function_t func) { + lsan_block_context_t *lsan_ctxt = alloc_lsan_context(ctxt, func); + return REAL(dispatch_after_f)(when, dq, (void *)lsan_ctxt, + lsan_dispatch_call_block_and_release); +} + +INTERCEPTOR(void, dispatch_group_async_f, dispatch_group_t group, + dispatch_queue_t dq, void *ctxt, dispatch_function_t func) { + lsan_block_context_t *lsan_ctxt = alloc_lsan_context(ctxt, func); + REAL(dispatch_group_async_f) + (group, dq, (void *)lsan_ctxt, lsan_dispatch_call_block_and_release); +} + +#if !defined(MISSING_BLOCKS_SUPPORT) +extern "C" { +void dispatch_async(dispatch_queue_t dq, void (^work)(void)); +void dispatch_group_async(dispatch_group_t dg, dispatch_queue_t dq, + void (^work)(void)); +void dispatch_after(dispatch_time_t when, dispatch_queue_t queue, + void (^work)(void)); +void dispatch_source_set_cancel_handler(dispatch_source_t ds, + void (^work)(void)); +void dispatch_source_set_event_handler(dispatch_source_t ds, + void (^work)(void)); +} + +#define GET_LSAN_BLOCK(work) \ + void (^lsan_block)(void); \ + int parent_tid = GetCurrentThread(); \ + lsan_block = ^(void) { \ + lsan_register_worker_thread(parent_tid); \ + work(); \ + } + +INTERCEPTOR(void, dispatch_async, dispatch_queue_t dq, void (^work)(void)) { + GET_LSAN_BLOCK(work); + REAL(dispatch_async)(dq, lsan_block); +} + +INTERCEPTOR(void, dispatch_group_async, dispatch_group_t dg, + dispatch_queue_t dq, void (^work)(void)) { + GET_LSAN_BLOCK(work); + REAL(dispatch_group_async)(dg, dq, lsan_block); +} + +INTERCEPTOR(void, dispatch_after, dispatch_time_t when, dispatch_queue_t queue, + void (^work)(void)) { + GET_LSAN_BLOCK(work); + REAL(dispatch_after)(when, queue, lsan_block); +} + +INTERCEPTOR(void, dispatch_source_set_cancel_handler, dispatch_source_t ds, + void (^work)(void)) { + if (!work) { + REAL(dispatch_source_set_cancel_handler)(ds, work); + return; + } + GET_LSAN_BLOCK(work); + REAL(dispatch_source_set_cancel_handler)(ds, lsan_block); +} + +INTERCEPTOR(void, dispatch_source_set_event_handler, dispatch_source_t ds, + void (^work)(void)) { + GET_LSAN_BLOCK(work); + REAL(dispatch_source_set_event_handler)(ds, lsan_block); +} +#endif + +#endif // SANITIZER_MAC diff --git a/contrib/compiler-rt/lib/lsan/lsan_malloc_mac.cc b/contrib/compiler-rt/lib/lsan/lsan_malloc_mac.cc new file mode 100644 index 0000000..9c1dacc --- /dev/null +++ b/contrib/compiler-rt/lib/lsan/lsan_malloc_mac.cc @@ -0,0 +1,55 @@ +//===-- lsan_malloc_mac.cc ------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of LeakSanitizer (LSan), a memory leak detector. +// +// Mac-specific malloc interception. +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" +#if SANITIZER_MAC + +#include "lsan.h" +#include "lsan_allocator.h" +#include "lsan_thread.h" + +using namespace __lsan; +#define COMMON_MALLOC_ZONE_NAME "lsan" +#define COMMON_MALLOC_ENTER() ENSURE_LSAN_INITED +#define COMMON_MALLOC_SANITIZER_INITIALIZED lsan_inited +#define COMMON_MALLOC_FORCE_LOCK() +#define COMMON_MALLOC_FORCE_UNLOCK() +#define COMMON_MALLOC_MEMALIGN(alignment, size) \ + GET_STACK_TRACE_MALLOC; \ + void *p = lsan_memalign(alignment, size, stack) +#define COMMON_MALLOC_MALLOC(size) \ + GET_STACK_TRACE_MALLOC; \ + void *p = lsan_malloc(size, stack) +#define COMMON_MALLOC_REALLOC(ptr, size) \ + GET_STACK_TRACE_MALLOC; \ + void *p = lsan_realloc(ptr, size, stack) +#define COMMON_MALLOC_CALLOC(count, size) \ + GET_STACK_TRACE_MALLOC; \ + void *p = lsan_calloc(count, size, stack) +#define COMMON_MALLOC_VALLOC(size) \ + GET_STACK_TRACE_MALLOC; \ + void *p = lsan_valloc(size, stack) +#define COMMON_MALLOC_FREE(ptr) \ + lsan_free(ptr) +#define COMMON_MALLOC_SIZE(ptr) \ + uptr size = lsan_mz_size(ptr) +#define COMMON_MALLOC_FILL_STATS(zone, stats) +#define COMMON_MALLOC_REPORT_UNKNOWN_REALLOC(ptr, zone_ptr, zone_name) \ + (void)zone_name; \ + Report("mz_realloc(%p) -- attempting to realloc unallocated memory.\n", ptr); +#define COMMON_MALLOC_NAMESPACE __lsan + +#include "sanitizer_common/sanitizer_malloc_mac.inc" + +#endif // SANITIZER_MAC diff --git a/contrib/compiler-rt/lib/lsan/lsan_thread.cc b/contrib/compiler-rt/lib/lsan/lsan_thread.cc index 5dff4f7..4404c8c 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_thread.cc +++ b/contrib/compiler-rt/lib/lsan/lsan_thread.cc @@ -19,13 +19,11 @@ #include "sanitizer_common/sanitizer_thread_registry.h" #include "sanitizer_common/sanitizer_tls_get_addr.h" #include "lsan_allocator.h" +#include "lsan_common.h" namespace __lsan { -const u32 kInvalidTid = (u32) -1; - static ThreadRegistry *thread_registry; -static THREADLOCAL u32 current_thread_tid = kInvalidTid; static ThreadContextBase *CreateThreadContext(u32 tid) { void *mem = MmapOrDie(sizeof(ThreadContext), "ThreadContext"); @@ -41,14 +39,6 @@ void InitializeThreadRegistry() { ThreadRegistry(CreateThreadContext, kMaxThreads, kThreadQuarantineSize); } -u32 GetCurrentThread() { - return current_thread_tid; -} - -void SetCurrentThread(u32 tid) { - current_thread_tid = tid; -} - ThreadContext::ThreadContext(int tid) : ThreadContextBase(tid), stack_begin_(0), @@ -87,7 +77,7 @@ u32 ThreadCreate(u32 parent_tid, uptr user_id, bool detached) { /* arg */ nullptr); } -void ThreadStart(u32 tid, uptr os_id) { +void ThreadStart(u32 tid, tid_t os_id, bool workerthread) { OnStartedArgs args; uptr stack_size = 0; uptr tls_size = 0; @@ -97,11 +87,12 @@ void ThreadStart(u32 tid, uptr os_id) { args.tls_end = args.tls_begin + tls_size; GetAllocatorCacheRange(&args.cache_begin, &args.cache_end); args.dtls = DTLS_Get(); - thread_registry->StartThread(tid, os_id, &args); + thread_registry->StartThread(tid, os_id, workerthread, &args); } void ThreadFinish() { thread_registry->FinishThread(GetCurrentThread()); + SetCurrentThread(kInvalidTid); } ThreadContext *CurrentThreadContext() { @@ -136,7 +127,7 @@ void EnsureMainThreadIDIsCorrect() { ///// Interface to the common LSan module. ///// -bool GetThreadRangesLocked(uptr os_id, uptr *stack_begin, uptr *stack_end, +bool GetThreadRangesLocked(tid_t os_id, uptr *stack_begin, uptr *stack_end, uptr *tls_begin, uptr *tls_end, uptr *cache_begin, uptr *cache_end, DTLS **dtls) { ThreadContext *context = static_cast<ThreadContext *>( @@ -152,7 +143,7 @@ bool GetThreadRangesLocked(uptr os_id, uptr *stack_begin, uptr *stack_end, return true; } -void ForEachExtraStackRange(uptr os_id, RangeIteratorCallback callback, +void ForEachExtraStackRange(tid_t os_id, RangeIteratorCallback callback, void *arg) { } diff --git a/contrib/compiler-rt/lib/lsan/lsan_thread.h b/contrib/compiler-rt/lib/lsan/lsan_thread.h index 10b7b57..b16d3d9 100644 --- a/contrib/compiler-rt/lib/lsan/lsan_thread.h +++ b/contrib/compiler-rt/lib/lsan/lsan_thread.h @@ -45,7 +45,7 @@ class ThreadContext : public ThreadContextBase { void InitializeThreadRegistry(); -void ThreadStart(u32 tid, uptr os_id); +void ThreadStart(u32 tid, tid_t os_id, bool workerthread = false); void ThreadFinish(); u32 ThreadCreate(u32 tid, uptr uid, bool detached); void ThreadJoin(u32 tid); diff --git a/contrib/compiler-rt/lib/lsan/weak_symbols.txt b/contrib/compiler-rt/lib/lsan/weak_symbols.txt new file mode 100644 index 0000000..da4f994 --- /dev/null +++ b/contrib/compiler-rt/lib/lsan/weak_symbols.txt @@ -0,0 +1,2 @@ +___lsan_default_suppressions +___lsan_is_turned_off diff --git a/contrib/compiler-rt/lib/msan/msan.h b/contrib/compiler-rt/lib/msan/msan.h index 0709260..fa9c15b 100644 --- a/contrib/compiler-rt/lib/msan/msan.h +++ b/contrib/compiler-rt/lib/msan/msan.h @@ -280,10 +280,18 @@ void InitializeInterceptors(); void MsanAllocatorInit(); void MsanAllocatorThreadFinish(); -void *MsanCalloc(StackTrace *stack, uptr nmemb, uptr size); -void *MsanReallocate(StackTrace *stack, void *oldp, uptr size, - uptr alignment, bool zeroise); void MsanDeallocate(StackTrace *stack, void *ptr); + +void *msan_malloc(uptr size, StackTrace *stack); +void *msan_calloc(uptr nmemb, uptr size, StackTrace *stack); +void *msan_realloc(void *ptr, uptr size, StackTrace *stack); +void *msan_valloc(uptr size, StackTrace *stack); +void *msan_pvalloc(uptr size, StackTrace *stack); +void *msan_aligned_alloc(uptr alignment, uptr size, StackTrace *stack); +void *msan_memalign(uptr alignment, uptr size, StackTrace *stack); +int msan_posix_memalign(void **memptr, uptr alignment, uptr size, + StackTrace *stack); + void InstallTrapHandler(); void InstallAtExitHandler(); diff --git a/contrib/compiler-rt/lib/msan/msan_allocator.cc b/contrib/compiler-rt/lib/msan/msan_allocator.cc index 6c389f0..1034dbdf 100644 --- a/contrib/compiler-rt/lib/msan/msan_allocator.cc +++ b/contrib/compiler-rt/lib/msan/msan_allocator.cc @@ -13,7 +13,9 @@ //===----------------------------------------------------------------------===// #include "sanitizer_common/sanitizer_allocator.h" +#include "sanitizer_common/sanitizer_allocator_checks.h" #include "sanitizer_common/sanitizer_allocator_interface.h" +#include "sanitizer_common/sanitizer_errno.h" #include "msan.h" #include "msan_allocator.h" #include "msan_origin.h" @@ -47,12 +49,18 @@ struct MsanMapUnmapCallback { static const uptr kRegionSizeLog = 20; static const uptr kNumRegions = SANITIZER_MMAP_RANGE_SIZE >> kRegionSizeLog; typedef TwoLevelByteMap<(kNumRegions >> 12), 1 << 12> ByteMap; - typedef CompactSizeClassMap SizeClassMap; - - typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, sizeof(Metadata), - SizeClassMap, kRegionSizeLog, ByteMap, - MsanMapUnmapCallback> PrimaryAllocator; + struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = sizeof(Metadata); + typedef __sanitizer::CompactSizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = __msan::kRegionSizeLog; + typedef __msan::ByteMap ByteMap; + typedef MsanMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; + }; + typedef SizeClassAllocator32<AP32> PrimaryAllocator; #elif defined(__x86_64__) #if SANITIZER_LINUX && !defined(MSAN_LINUX_X86_64_OLD_MAPPING) static const uptr kAllocatorSpace = 0x700000000000ULL; @@ -90,11 +98,18 @@ struct MsanMapUnmapCallback { static const uptr kRegionSizeLog = 20; static const uptr kNumRegions = SANITIZER_MMAP_RANGE_SIZE >> kRegionSizeLog; typedef TwoLevelByteMap<(kNumRegions >> 12), 1 << 12> ByteMap; - typedef CompactSizeClassMap SizeClassMap; - typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, sizeof(Metadata), - SizeClassMap, kRegionSizeLog, ByteMap, - MsanMapUnmapCallback> PrimaryAllocator; + struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = sizeof(Metadata); + typedef __sanitizer::CompactSizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = __msan::kRegionSizeLog; + typedef __msan::ByteMap ByteMap; + typedef MsanMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; + }; + typedef SizeClassAllocator32<AP32> PrimaryAllocator; #endif typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache; typedef LargeMmapAllocator<MsanMapUnmapCallback> SecondaryAllocator; @@ -106,9 +121,8 @@ static AllocatorCache fallback_allocator_cache; static SpinMutex fallback_mutex; void MsanAllocatorInit() { - allocator.Init( - common_flags()->allocator_may_return_null, - common_flags()->allocator_release_to_os_interval_ms); + SetAllocatorMayReturnNull(common_flags()->allocator_may_return_null); + allocator.Init(common_flags()->allocator_release_to_os_interval_ms); } AllocatorCache *GetAllocatorCache(MsanThreadLocalMallocStorage *ms) { @@ -126,17 +140,17 @@ static void *MsanAllocate(StackTrace *stack, uptr size, uptr alignment, if (size > kMaxAllowedMallocSize) { Report("WARNING: MemorySanitizer failed to allocate %p bytes\n", (void *)size); - return allocator.ReturnNullOrDieOnBadRequest(); + return Allocator::FailureHandler::OnBadRequest(); } MsanThread *t = GetCurrentThread(); void *allocated; if (t) { AllocatorCache *cache = GetAllocatorCache(&t->malloc_storage()); - allocated = allocator.Allocate(cache, size, alignment, false); + allocated = allocator.Allocate(cache, size, alignment); } else { SpinMutexLock l(&fallback_mutex); AllocatorCache *cache = &fallback_allocator_cache; - allocated = allocator.Allocate(cache, size, alignment, false); + allocated = allocator.Allocate(cache, size, alignment); } Metadata *meta = reinterpret_cast<Metadata *>(allocator.GetMetaData(allocated)); @@ -182,20 +196,8 @@ void MsanDeallocate(StackTrace *stack, void *p) { } } -void *MsanCalloc(StackTrace *stack, uptr nmemb, uptr size) { - if (CallocShouldReturnNullDueToOverflow(size, nmemb)) - return allocator.ReturnNullOrDieOnBadRequest(); - return MsanReallocate(stack, nullptr, nmemb * size, sizeof(u64), true); -} - void *MsanReallocate(StackTrace *stack, void *old_p, uptr new_size, - uptr alignment, bool zeroise) { - if (!old_p) - return MsanAllocate(stack, new_size, alignment, zeroise); - if (!new_size) { - MsanDeallocate(stack, old_p); - return nullptr; - } + uptr alignment) { Metadata *meta = reinterpret_cast<Metadata*>(allocator.GetMetaData(old_p)); uptr old_size = meta->requested_size; uptr actually_allocated_size = allocator.GetActuallyAllocatedSize(old_p); @@ -203,10 +205,7 @@ void *MsanReallocate(StackTrace *stack, void *old_p, uptr new_size, // We are not reallocating here. meta->requested_size = new_size; if (new_size > old_size) { - if (zeroise) { - __msan_clear_and_unpoison((char *)old_p + old_size, - new_size - old_size); - } else if (flags()->poison_in_malloc) { + if (flags()->poison_in_malloc) { stack->tag = StackTrace::TAG_ALLOC; PoisonMemory((char *)old_p + old_size, new_size - old_size, stack); } @@ -214,8 +213,7 @@ void *MsanReallocate(StackTrace *stack, void *old_p, uptr new_size, return old_p; } uptr memcpy_size = Min(new_size, old_size); - void *new_p = MsanAllocate(stack, new_size, alignment, zeroise); - // Printf("realloc: old_size %zd new_size %zd\n", old_size, new_size); + void *new_p = MsanAllocate(stack, new_size, alignment, false /*zeroise*/); if (new_p) { CopyMemory(new_p, old_p, memcpy_size, stack); MsanDeallocate(stack, old_p); @@ -231,6 +229,67 @@ static uptr AllocationSize(const void *p) { return b->requested_size; } +void *msan_malloc(uptr size, StackTrace *stack) { + return SetErrnoOnNull(MsanAllocate(stack, size, sizeof(u64), false)); +} + +void *msan_calloc(uptr nmemb, uptr size, StackTrace *stack) { + if (UNLIKELY(CheckForCallocOverflow(size, nmemb))) + return SetErrnoOnNull(Allocator::FailureHandler::OnBadRequest()); + return SetErrnoOnNull(MsanAllocate(stack, nmemb * size, sizeof(u64), true)); +} + +void *msan_realloc(void *ptr, uptr size, StackTrace *stack) { + if (!ptr) + return SetErrnoOnNull(MsanAllocate(stack, size, sizeof(u64), false)); + if (size == 0) { + MsanDeallocate(stack, ptr); + return nullptr; + } + return SetErrnoOnNull(MsanReallocate(stack, ptr, size, sizeof(u64))); +} + +void *msan_valloc(uptr size, StackTrace *stack) { + return SetErrnoOnNull(MsanAllocate(stack, size, GetPageSizeCached(), false)); +} + +void *msan_pvalloc(uptr size, StackTrace *stack) { + uptr PageSize = GetPageSizeCached(); + // pvalloc(0) should allocate one page. + size = size == 0 ? PageSize : RoundUpTo(size, PageSize); + return SetErrnoOnNull(MsanAllocate(stack, size, PageSize, false)); +} + +void *msan_aligned_alloc(uptr alignment, uptr size, StackTrace *stack) { + if (UNLIKELY(!CheckAlignedAllocAlignmentAndSize(alignment, size))) { + errno = errno_EINVAL; + return Allocator::FailureHandler::OnBadRequest(); + } + return SetErrnoOnNull(MsanAllocate(stack, size, alignment, false)); +} + +void *msan_memalign(uptr alignment, uptr size, StackTrace *stack) { + if (UNLIKELY(!IsPowerOfTwo(alignment))) { + errno = errno_EINVAL; + return Allocator::FailureHandler::OnBadRequest(); + } + return SetErrnoOnNull(MsanAllocate(stack, size, alignment, false)); +} + +int msan_posix_memalign(void **memptr, uptr alignment, uptr size, + StackTrace *stack) { + if (UNLIKELY(!CheckPosixMemalignAlignment(alignment))) { + Allocator::FailureHandler::OnBadRequest(); + return errno_EINVAL; + } + void *ptr = MsanAllocate(stack, size, alignment, false); + if (UNLIKELY(!ptr)) + return errno_ENOMEM; + CHECK(IsAligned((uptr)ptr, alignment)); + *memptr = ptr; + return 0; +} + } // namespace __msan using namespace __msan; diff --git a/contrib/compiler-rt/lib/msan/msan_interceptors.cc b/contrib/compiler-rt/lib/msan/msan_interceptors.cc index 6447bb1..b5d22ba 100644 --- a/contrib/compiler-rt/lib/msan/msan_interceptors.cc +++ b/contrib/compiler-rt/lib/msan/msan_interceptors.cc @@ -27,6 +27,7 @@ #include "sanitizer_common/sanitizer_allocator_internal.h" #include "sanitizer_common/sanitizer_atomic.h" #include "sanitizer_common/sanitizer_common.h" +#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_stackdepot.h" #include "sanitizer_common/sanitizer_libc.h" #include "sanitizer_common/sanitizer_linux.h" @@ -48,15 +49,9 @@ DECLARE_REAL(SIZE_T, strnlen, const char *s, SIZE_T maxlen) DECLARE_REAL(void *, memcpy, void *dest, const void *src, uptr n) DECLARE_REAL(void *, memset, void *dest, int c, uptr n) -#if SANITIZER_FREEBSD -#define __errno_location __error -#endif - // True if this is a nested interceptor. static THREADLOCAL int in_interceptor_scope; -extern "C" int *__errno_location(void); - struct InterceptorScope { InterceptorScope() { ++in_interceptor_scope; } ~InterceptorScope() { --in_interceptor_scope; } @@ -123,14 +118,6 @@ static void *AllocateFromLocalPool(uptr size_in_bytes) { #define CHECK_UNPOISONED_STRING(x, n) \ CHECK_UNPOISONED_STRING_OF_LEN((x), internal_strlen(x), (n)) -INTERCEPTOR(SIZE_T, fread, void *ptr, SIZE_T size, SIZE_T nmemb, void *file) { - ENSURE_MSAN_INITED(); - SIZE_T res = REAL(fread)(ptr, size, nmemb, file); - if (res > 0) - __msan_unpoison(ptr, res *size); - return res; -} - #if !SANITIZER_FREEBSD INTERCEPTOR(SIZE_T, fread_unlocked, void *ptr, SIZE_T size, SIZE_T nmemb, void *file) { @@ -174,58 +161,45 @@ INTERCEPTOR(void *, bcopy, const void *src, void *dest, SIZE_T n) { INTERCEPTOR(int, posix_memalign, void **memptr, SIZE_T alignment, SIZE_T size) { GET_MALLOC_STACK_TRACE; - CHECK_EQ(alignment & (alignment - 1), 0); CHECK_NE(memptr, 0); - *memptr = MsanReallocate(&stack, nullptr, size, alignment, false); - CHECK_NE(*memptr, 0); - __msan_unpoison(memptr, sizeof(*memptr)); - return 0; + int res = msan_posix_memalign(memptr, alignment, size, &stack); + if (!res) + __msan_unpoison(memptr, sizeof(*memptr)); + return res; } #if !SANITIZER_FREEBSD -INTERCEPTOR(void *, memalign, SIZE_T boundary, SIZE_T size) { +INTERCEPTOR(void *, memalign, SIZE_T alignment, SIZE_T size) { GET_MALLOC_STACK_TRACE; - CHECK_EQ(boundary & (boundary - 1), 0); - void *ptr = MsanReallocate(&stack, nullptr, size, boundary, false); - return ptr; + return msan_memalign(alignment, size, &stack); } #define MSAN_MAYBE_INTERCEPT_MEMALIGN INTERCEPT_FUNCTION(memalign) #else #define MSAN_MAYBE_INTERCEPT_MEMALIGN #endif -INTERCEPTOR(void *, aligned_alloc, SIZE_T boundary, SIZE_T size) { +INTERCEPTOR(void *, aligned_alloc, SIZE_T alignment, SIZE_T size) { GET_MALLOC_STACK_TRACE; - CHECK_EQ(boundary & (boundary - 1), 0); - void *ptr = MsanReallocate(&stack, nullptr, size, boundary, false); - return ptr; + return msan_aligned_alloc(alignment, size, &stack); } -INTERCEPTOR(void *, __libc_memalign, SIZE_T boundary, SIZE_T size) { +INTERCEPTOR(void *, __libc_memalign, SIZE_T alignment, SIZE_T size) { GET_MALLOC_STACK_TRACE; - CHECK_EQ(boundary & (boundary - 1), 0); - void *ptr = MsanReallocate(&stack, nullptr, size, boundary, false); - DTLS_on_libc_memalign(ptr, size); + void *ptr = msan_memalign(alignment, size, &stack); + if (ptr) + DTLS_on_libc_memalign(ptr, size); return ptr; } INTERCEPTOR(void *, valloc, SIZE_T size) { GET_MALLOC_STACK_TRACE; - void *ptr = MsanReallocate(&stack, nullptr, size, GetPageSizeCached(), false); - return ptr; + return msan_valloc(size, &stack); } #if !SANITIZER_FREEBSD INTERCEPTOR(void *, pvalloc, SIZE_T size) { GET_MALLOC_STACK_TRACE; - uptr PageSize = GetPageSizeCached(); - size = RoundUpTo(size, PageSize); - if (size == 0) { - // pvalloc(0) should allocate one page. - size = PageSize; - } - void *ptr = MsanReallocate(&stack, nullptr, size, PageSize, false); - return ptr; + return msan_pvalloc(size, &stack); } #define MSAN_MAYBE_INTERCEPT_PVALLOC INTERCEPT_FUNCTION(pvalloc) #else @@ -349,33 +323,6 @@ INTERCEPTOR(char *, __strdup, char *src) { #define MSAN_MAYBE_INTERCEPT___STRDUP #endif -INTERCEPTOR(char *, strndup, char *src, SIZE_T n) { - ENSURE_MSAN_INITED(); - GET_STORE_STACK_TRACE; - // On FreeBSD strndup() leverages strnlen(). - InterceptorScope interceptor_scope; - SIZE_T copy_size = REAL(strnlen)(src, n); - char *res = REAL(strndup)(src, n); - CopyShadowAndOrigin(res, src, copy_size, &stack); - __msan_unpoison(res + copy_size, 1); // \0 - return res; -} - -#if !SANITIZER_FREEBSD -INTERCEPTOR(char *, __strndup, char *src, SIZE_T n) { - ENSURE_MSAN_INITED(); - GET_STORE_STACK_TRACE; - SIZE_T copy_size = REAL(strnlen)(src, n); - char *res = REAL(__strndup)(src, n); - CopyShadowAndOrigin(res, src, copy_size, &stack); - __msan_unpoison(res + copy_size, 1); // \0 - return res; -} -#define MSAN_MAYBE_INTERCEPT___STRNDUP INTERCEPT_FUNCTION(__strndup) -#else -#define MSAN_MAYBE_INTERCEPT___STRNDUP -#endif - INTERCEPTOR(char *, gcvt, double number, SIZE_T ndigit, char *buf) { ENSURE_MSAN_INITED(); char *res = REAL(gcvt)(number, ndigit, buf); @@ -573,30 +520,6 @@ INTERCEPTOR(int, mbrtowc, wchar_t *dest, const char *src, SIZE_T n, void *ps) { return res; } -INTERCEPTOR(SIZE_T, wcslen, const wchar_t *s) { - ENSURE_MSAN_INITED(); - SIZE_T res = REAL(wcslen)(s); - CHECK_UNPOISONED(s, sizeof(wchar_t) * (res + 1)); - return res; -} - -// wchar_t *wcschr(const wchar_t *wcs, wchar_t wc); -INTERCEPTOR(wchar_t *, wcschr, void *s, wchar_t wc, void *ps) { - ENSURE_MSAN_INITED(); - wchar_t *res = REAL(wcschr)(s, wc, ps); - return res; -} - -// wchar_t *wcscpy(wchar_t *dest, const wchar_t *src); -INTERCEPTOR(wchar_t *, wcscpy, wchar_t *dest, const wchar_t *src) { - ENSURE_MSAN_INITED(); - GET_STORE_STACK_TRACE; - wchar_t *res = REAL(wcscpy)(dest, src); - CopyShadowAndOrigin(dest, src, sizeof(wchar_t) * (REAL(wcslen)(src) + 1), - &stack); - return res; -} - // wchar_t *wmemcpy(wchar_t *dest, const wchar_t *src, SIZE_T n); INTERCEPTOR(wchar_t *, wmemcpy, wchar_t *dest, const wchar_t *src, SIZE_T n) { ENSURE_MSAN_INITED(); @@ -917,7 +840,7 @@ INTERCEPTOR(void *, calloc, SIZE_T nmemb, SIZE_T size) { if (UNLIKELY(!msan_inited)) // Hack: dlsym calls calloc before REAL(calloc) is retrieved from dlsym. return AllocateFromLocalPool(nmemb * size); - return MsanCalloc(&stack, nmemb, size); + return msan_calloc(nmemb, size, &stack); } INTERCEPTOR(void *, realloc, void *ptr, SIZE_T size) { @@ -930,12 +853,12 @@ INTERCEPTOR(void *, realloc, void *ptr, SIZE_T size) { new_ptr = AllocateFromLocalPool(copy_size); } else { copy_size = size; - new_ptr = MsanReallocate(&stack, nullptr, copy_size, sizeof(u64), false); + new_ptr = msan_malloc(copy_size, &stack); } internal_memcpy(new_ptr, ptr, copy_size); return new_ptr; } - return MsanReallocate(&stack, ptr, size, sizeof(u64), false); + return msan_realloc(ptr, size, &stack); } INTERCEPTOR(void *, malloc, SIZE_T size) { @@ -943,7 +866,7 @@ INTERCEPTOR(void *, malloc, SIZE_T size) { if (UNLIKELY(!msan_inited)) // Hack: dlsym calls malloc before REAL(malloc) is retrieved from dlsym. return AllocateFromLocalPool(size); - return MsanReallocate(&stack, nullptr, size, sizeof(u64), false); + return msan_malloc(size, &stack); } void __msan_allocated_memory(const void *data, uptr size) { @@ -974,7 +897,7 @@ INTERCEPTOR(void *, mmap, void *addr, SIZE_T length, int prot, int flags, ENSURE_MSAN_INITED(); if (addr && !MEM_IS_APP(addr)) { if (flags & map_fixed) { - *__errno_location() = errno_EINVAL; + errno = errno_EINVAL; return (void *)-1; } else { addr = nullptr; @@ -992,7 +915,7 @@ INTERCEPTOR(void *, mmap64, void *addr, SIZE_T length, int prot, int flags, ENSURE_MSAN_INITED(); if (addr && !MEM_IS_APP(addr)) { if (flags & map_fixed) { - *__errno_location() = errno_EINVAL; + errno = errno_EINVAL; return (void *)-1; } else { addr = nullptr; @@ -1360,6 +1283,13 @@ int OnExit() { return __msan_memcpy(to, from, size); \ } +#define COMMON_INTERCEPTOR_COPY_STRING(ctx, to, from, size) \ + do { \ + GET_STORE_STACK_TRACE; \ + CopyShadowAndOrigin(to, from, size, &stack); \ + __msan_unpoison(to + size, 1); \ + } while (false) + #include "sanitizer_common/sanitizer_platform_interceptors.h" #include "sanitizer_common/sanitizer_common_interceptors.inc" @@ -1433,6 +1363,35 @@ INTERCEPTOR(int, dl_iterate_phdr, dl_iterate_phdr_cb callback, void *data) { return res; } +// wchar_t *wcschr(const wchar_t *wcs, wchar_t wc); +INTERCEPTOR(wchar_t *, wcschr, void *s, wchar_t wc, void *ps) { + ENSURE_MSAN_INITED(); + wchar_t *res = REAL(wcschr)(s, wc, ps); + return res; +} + +// wchar_t *wcscpy(wchar_t *dest, const wchar_t *src); +INTERCEPTOR(wchar_t *, wcscpy, wchar_t *dest, const wchar_t *src) { + ENSURE_MSAN_INITED(); + GET_STORE_STACK_TRACE; + wchar_t *res = REAL(wcscpy)(dest, src); + CopyShadowAndOrigin(dest, src, sizeof(wchar_t) * (REAL(wcslen)(src) + 1), + &stack); + return res; +} + +INTERCEPTOR(wchar_t *, wcsncpy, wchar_t *dest, const wchar_t *src, + SIZE_T n) { // NOLINT + ENSURE_MSAN_INITED(); + GET_STORE_STACK_TRACE; + SIZE_T copy_size = REAL(wcsnlen)(src, n); + if (copy_size < n) copy_size++; // trailing \0 + wchar_t *res = REAL(wcsncpy)(dest, src, n); // NOLINT + CopyShadowAndOrigin(dest, src, copy_size * sizeof(wchar_t), &stack); + __msan_unpoison(dest + copy_size, (n - copy_size) * sizeof(wchar_t)); + return res; +} + // These interface functions reside here so that they can use // REAL(memset), etc. void __msan_unpoison(const void *a, uptr size) { @@ -1527,8 +1486,6 @@ void InitializeInterceptors() { INTERCEPT_FUNCTION(stpcpy); // NOLINT INTERCEPT_FUNCTION(strdup); MSAN_MAYBE_INTERCEPT___STRDUP; - INTERCEPT_FUNCTION(strndup); - MSAN_MAYBE_INTERCEPT___STRNDUP; INTERCEPT_FUNCTION(strncpy); // NOLINT INTERCEPT_FUNCTION(gcvt); INTERCEPT_FUNCTION(strcat); // NOLINT @@ -1565,8 +1522,10 @@ void InitializeInterceptors() { INTERCEPT_FUNCTION(mbtowc); INTERCEPT_FUNCTION(mbrtowc); INTERCEPT_FUNCTION(wcslen); + INTERCEPT_FUNCTION(wcsnlen); INTERCEPT_FUNCTION(wcschr); INTERCEPT_FUNCTION(wcscpy); + INTERCEPT_FUNCTION(wcsncpy); INTERCEPT_FUNCTION(wcscmp); INTERCEPT_FUNCTION(getenv); INTERCEPT_FUNCTION(setenv); diff --git a/contrib/compiler-rt/lib/msan/msan_new_delete.cc b/contrib/compiler-rt/lib/msan/msan_new_delete.cc index 5401003..7219267 100644 --- a/contrib/compiler-rt/lib/msan/msan_new_delete.cc +++ b/contrib/compiler-rt/lib/msan/msan_new_delete.cc @@ -14,6 +14,7 @@ #include "msan.h" #include "interception/interception.h" +#include "sanitizer_common/sanitizer_allocator.h" #if MSAN_REPLACE_OPERATORS_NEW_AND_DELETE @@ -27,18 +28,25 @@ namespace std { } // namespace std -#define OPERATOR_NEW_BODY \ +// TODO(alekseys): throw std::bad_alloc instead of dying on OOM. +#define OPERATOR_NEW_BODY(nothrow) \ GET_MALLOC_STACK_TRACE; \ - return MsanReallocate(&stack, 0, size, sizeof(u64), false) + void *res = msan_malloc(size, &stack);\ + if (!nothrow && UNLIKELY(!res)) DieOnFailure::OnOOM();\ + return res INTERCEPTOR_ATTRIBUTE -void *operator new(size_t size) { OPERATOR_NEW_BODY; } +void *operator new(size_t size) { OPERATOR_NEW_BODY(false /*nothrow*/); } INTERCEPTOR_ATTRIBUTE -void *operator new[](size_t size) { OPERATOR_NEW_BODY; } +void *operator new[](size_t size) { OPERATOR_NEW_BODY(false /*nothrow*/); } INTERCEPTOR_ATTRIBUTE -void *operator new(size_t size, std::nothrow_t const&) { OPERATOR_NEW_BODY; } +void *operator new(size_t size, std::nothrow_t const&) { + OPERATOR_NEW_BODY(true /*nothrow*/); +} INTERCEPTOR_ATTRIBUTE -void *operator new[](size_t size, std::nothrow_t const&) { OPERATOR_NEW_BODY; } +void *operator new[](size_t size, std::nothrow_t const&) { + OPERATOR_NEW_BODY(true /*nothrow*/); +} #define OPERATOR_DELETE_BODY \ GET_MALLOC_STACK_TRACE; \ diff --git a/contrib/compiler-rt/lib/profile/InstrProfData.inc b/contrib/compiler-rt/lib/profile/InstrProfData.inc index f7c22d1..be0dd4a 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfData.inc +++ b/contrib/compiler-rt/lib/profile/InstrProfData.inc @@ -153,7 +153,17 @@ INSTR_PROF_RAW_HEADER(uint64_t, ValueKindLast, IPVK_Last) VALUE_PROF_FUNC_PARAM(uint64_t, TargetValue, Type::getInt64Ty(Ctx)) \ INSTR_PROF_COMMA VALUE_PROF_FUNC_PARAM(void *, Data, Type::getInt8PtrTy(Ctx)) INSTR_PROF_COMMA +#ifndef VALUE_RANGE_PROF VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx)) +#else /* VALUE_RANGE_PROF */ +VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx)) \ + INSTR_PROF_COMMA +VALUE_PROF_FUNC_PARAM(uint64_t, PreciseRangeStart, Type::getInt64Ty(Ctx)) \ + INSTR_PROF_COMMA +VALUE_PROF_FUNC_PARAM(uint64_t, PreciseRangeLast, Type::getInt64Ty(Ctx)) \ + INSTR_PROF_COMMA +VALUE_PROF_FUNC_PARAM(uint64_t, LargeValue, Type::getInt64Ty(Ctx)) +#endif /*VALUE_RANGE_PROF */ #undef VALUE_PROF_FUNC_PARAM #undef INSTR_PROF_COMMA /* VALUE_PROF_FUNC_PARAM end */ @@ -174,13 +184,15 @@ VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx)) * name hash and the function address. */ VALUE_PROF_KIND(IPVK_IndirectCallTarget, 0) +/* For memory intrinsic functions size profiling. */ +VALUE_PROF_KIND(IPVK_MemOPSize, 1) /* These two kinds must be the last to be * declared. This is to make sure the string * array created with the template can be * indexed with the kind value. */ VALUE_PROF_KIND(IPVK_First, IPVK_IndirectCallTarget) -VALUE_PROF_KIND(IPVK_Last, IPVK_IndirectCallTarget) +VALUE_PROF_KIND(IPVK_Last, IPVK_MemOPSize) #undef VALUE_PROF_KIND /* VALUE_PROF_KIND end */ @@ -234,6 +246,31 @@ COVMAP_HEADER(uint32_t, Int32Ty, Version, \ /* COVMAP_HEADER end. */ +#ifdef INSTR_PROF_SECT_ENTRY +#define INSTR_PROF_DATA_DEFINED +INSTR_PROF_SECT_ENTRY(IPSK_data, \ + INSTR_PROF_QUOTE(INSTR_PROF_DATA_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_DATA_COFF), "__DATA,") +INSTR_PROF_SECT_ENTRY(IPSK_cnts, \ + INSTR_PROF_QUOTE(INSTR_PROF_CNTS_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_CNTS_COFF), "__DATA,") +INSTR_PROF_SECT_ENTRY(IPSK_name, \ + INSTR_PROF_QUOTE(INSTR_PROF_NAME_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_NAME_COFF), "__DATA,") +INSTR_PROF_SECT_ENTRY(IPSK_vals, \ + INSTR_PROF_QUOTE(INSTR_PROF_VALS_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_VALS_COFF), "__DATA,") +INSTR_PROF_SECT_ENTRY(IPSK_vnodes, \ + INSTR_PROF_QUOTE(INSTR_PROF_VNODES_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_VNODES_COFF), "__DATA,") +INSTR_PROF_SECT_ENTRY(IPSK_covmap, \ + INSTR_PROF_QUOTE(INSTR_PROF_COVMAP_COMMON), \ + INSTR_PROF_QUOTE(INSTR_PROF_COVMAP_COFF), "__LLVM_COV,") + +#undef INSTR_PROF_SECT_ENTRY +#endif + + #ifdef INSTR_PROF_VALUE_PROF_DATA #define INSTR_PROF_DATA_DEFINED @@ -610,17 +647,47 @@ serializeValueProfDataFrom(ValueProfRecordClosure *Closure, * specified via command line. */ #define INSTR_PROF_PROFILE_NAME_VAR __llvm_profile_filename +/* section name strings common to all targets other + than WIN32 */ +#define INSTR_PROF_DATA_COMMON __llvm_prf_data +#define INSTR_PROF_NAME_COMMON __llvm_prf_names +#define INSTR_PROF_CNTS_COMMON __llvm_prf_cnts +#define INSTR_PROF_VALS_COMMON __llvm_prf_vals +#define INSTR_PROF_VNODES_COMMON __llvm_prf_vnds +#define INSTR_PROF_COVMAP_COMMON __llvm_covmap +/* Win32 */ +#define INSTR_PROF_DATA_COFF .lprfd +#define INSTR_PROF_NAME_COFF .lprfn +#define INSTR_PROF_CNTS_COFF .lprfc +#define INSTR_PROF_VALS_COFF .lprfv +#define INSTR_PROF_VNODES_COFF .lprfnd +#define INSTR_PROF_COVMAP_COFF .lcovmap + +#ifdef _WIN32 /* Runtime section names and name strings. */ -#define INSTR_PROF_DATA_SECT_NAME __llvm_prf_data -#define INSTR_PROF_NAME_SECT_NAME __llvm_prf_names -#define INSTR_PROF_CNTS_SECT_NAME __llvm_prf_cnts +#define INSTR_PROF_DATA_SECT_NAME INSTR_PROF_DATA_COFF +#define INSTR_PROF_NAME_SECT_NAME INSTR_PROF_NAME_COFF +#define INSTR_PROF_CNTS_SECT_NAME INSTR_PROF_CNTS_COFF /* Array of pointers. Each pointer points to a list * of value nodes associated with one value site. */ -#define INSTR_PROF_VALS_SECT_NAME __llvm_prf_vals +#define INSTR_PROF_VALS_SECT_NAME INSTR_PROF_VALS_COFF /* Value profile nodes section. */ -#define INSTR_PROF_VNODES_SECT_NAME __llvm_prf_vnds -#define INSTR_PROF_COVMAP_SECT_NAME __llvm_covmap +#define INSTR_PROF_VNODES_SECT_NAME INSTR_PROF_VNODES_COFF +#define INSTR_PROF_COVMAP_SECT_NAME INSTR_PROF_COVMAP_COFF +#else +/* Runtime section names and name strings. */ +#define INSTR_PROF_DATA_SECT_NAME INSTR_PROF_DATA_COMMON +#define INSTR_PROF_NAME_SECT_NAME INSTR_PROF_NAME_COMMON +#define INSTR_PROF_CNTS_SECT_NAME INSTR_PROF_CNTS_COMMON +/* Array of pointers. Each pointer points to a list + * of value nodes associated with one value site. + */ +#define INSTR_PROF_VALS_SECT_NAME INSTR_PROF_VALS_COMMON +/* Value profile nodes section. */ +#define INSTR_PROF_VNODES_SECT_NAME INSTR_PROF_VNODES_COMMON +#define INSTR_PROF_COVMAP_SECT_NAME INSTR_PROF_COVMAP_COMMON +#endif #define INSTR_PROF_DATA_SECT_NAME_STR \ INSTR_PROF_QUOTE(INSTR_PROF_DATA_SECT_NAME) @@ -649,6 +716,9 @@ serializeValueProfDataFrom(ValueProfRecordClosure *Closure, #define INSTR_PROF_VALUE_PROF_FUNC __llvm_profile_instrument_target #define INSTR_PROF_VALUE_PROF_FUNC_STR \ INSTR_PROF_QUOTE(INSTR_PROF_VALUE_PROF_FUNC) +#define INSTR_PROF_VALUE_RANGE_PROF_FUNC __llvm_profile_instrument_range +#define INSTR_PROF_VALUE_RANGE_PROF_FUNC_STR \ + INSTR_PROF_QUOTE(INSTR_PROF_VALUE_RANGE_PROF_FUNC) /* InstrProfile per-function control data alignment. */ #define INSTR_PROF_DATA_ALIGNMENT 8 diff --git a/contrib/compiler-rt/lib/profile/InstrProfiling.c b/contrib/compiler-rt/lib/profile/InstrProfiling.c index 6828a3d..fe66fec 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfiling.c +++ b/contrib/compiler-rt/lib/profile/InstrProfiling.c @@ -19,8 +19,6 @@ COMPILER_RT_WEAK uint64_t INSTR_PROF_RAW_VERSION_VAR = INSTR_PROF_RAW_VERSION; -COMPILER_RT_WEAK char INSTR_PROF_PROFILE_NAME_VAR[1] = {0}; - COMPILER_RT_VISIBILITY uint64_t __llvm_profile_get_magic(void) { return sizeof(void *) == sizeof(uint64_t) ? (INSTR_PROF_RAW_MAGIC_64) : (INSTR_PROF_RAW_MAGIC_32); diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingBuffer.c b/contrib/compiler-rt/lib/profile/InstrProfilingBuffer.c index ac259e8..a7e852f 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingBuffer.c +++ b/contrib/compiler-rt/lib/profile/InstrProfilingBuffer.c @@ -45,15 +45,24 @@ uint64_t __llvm_profile_get_size_for_buffer_internal( (CountersEnd - CountersBegin) * sizeof(uint64_t) + NamesSize + Padding; } +COMPILER_RT_VISIBILITY +void initBufferWriter(ProfDataWriter *BufferWriter, char *Buffer) { + BufferWriter->Write = lprofBufferWriter; + BufferWriter->WriterCtx = Buffer; +} + COMPILER_RT_VISIBILITY int __llvm_profile_write_buffer(char *Buffer) { - return lprofWriteData(lprofBufferWriter, Buffer, 0); + ProfDataWriter BufferWriter; + initBufferWriter(&BufferWriter, Buffer); + return lprofWriteData(&BufferWriter, 0, 0); } COMPILER_RT_VISIBILITY int __llvm_profile_write_buffer_internal( char *Buffer, const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd, const uint64_t *CountersBegin, const uint64_t *CountersEnd, const char *NamesBegin, const char *NamesEnd) { - return lprofWriteDataImpl(lprofBufferWriter, Buffer, DataBegin, DataEnd, - CountersBegin, CountersEnd, 0, NamesBegin, - NamesEnd); + ProfDataWriter BufferWriter; + initBufferWriter(&BufferWriter, Buffer); + return lprofWriteDataImpl(&BufferWriter, DataBegin, DataEnd, CountersBegin, + CountersEnd, 0, NamesBegin, NamesEnd, 0); } diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingFile.c b/contrib/compiler-rt/lib/profile/InstrProfilingFile.c index f82080c..d038bb9 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingFile.c +++ b/contrib/compiler-rt/lib/profile/InstrProfilingFile.c @@ -91,24 +91,39 @@ static const char *getCurFilename(char *FilenameBuf); static unsigned doMerging() { return lprofCurFilename.MergePoolSize; } /* Return 1 if there is an error, otherwise return 0. */ -static uint32_t fileWriter(ProfDataIOVec *IOVecs, uint32_t NumIOVecs, - void **WriterCtx) { +static uint32_t fileWriter(ProfDataWriter *This, ProfDataIOVec *IOVecs, + uint32_t NumIOVecs) { uint32_t I; - FILE *File = (FILE *)*WriterCtx; + FILE *File = (FILE *)This->WriterCtx; for (I = 0; I < NumIOVecs; I++) { - if (fwrite(IOVecs[I].Data, IOVecs[I].ElmSize, IOVecs[I].NumElm, File) != - IOVecs[I].NumElm) - return 1; + if (IOVecs[I].Data) { + if (fwrite(IOVecs[I].Data, IOVecs[I].ElmSize, IOVecs[I].NumElm, File) != + IOVecs[I].NumElm) + return 1; + } else { + if (fseek(File, IOVecs[I].ElmSize * IOVecs[I].NumElm, SEEK_CUR) == -1) + return 1; + } } return 0; } +static void initFileWriter(ProfDataWriter *This, FILE *File) { + This->Write = fileWriter; + This->WriterCtx = File; +} + COMPILER_RT_VISIBILITY ProfBufferIO * lprofCreateBufferIOInternal(void *File, uint32_t BufferSz) { FreeHook = &free; DynamicBufferIOBuffer = (uint8_t *)calloc(BufferSz, 1); VPBufferSize = BufferSz; - return lprofCreateBufferIO(fileWriter, File); + ProfDataWriter *fileWriter = + (ProfDataWriter *)calloc(sizeof(ProfDataWriter), 1); + initFileWriter(fileWriter, File); + ProfBufferIO *IO = lprofCreateBufferIO(fileWriter); + IO->OwnFileWriter = 1; + return IO; } static void setupIOBuffer() { @@ -122,9 +137,10 @@ static void setupIOBuffer() { /* Read profile data in \c ProfileFile and merge with in-memory profile counters. Returns -1 if there is fatal error, otheriwse - 0 is returned. + 0 is returned. Returning 0 does not mean merge is actually + performed. If merge is actually done, *MergeDone is set to 1. */ -static int doProfileMerging(FILE *ProfileFile) { +static int doProfileMerging(FILE *ProfileFile, int *MergeDone) { uint64_t ProfileFileSize; char *ProfileBuffer; @@ -169,9 +185,21 @@ static int doProfileMerging(FILE *ProfileFile) { __llvm_profile_merge_from_buffer(ProfileBuffer, ProfileFileSize); (void)munmap(ProfileBuffer, ProfileFileSize); + *MergeDone = 1; + return 0; } +/* Create the directory holding the file, if needed. */ +static void createProfileDir(const char *Filename) { + size_t Length = strlen(Filename); + if (lprofFindFirstDirSeparator(Filename)) { + char *Copy = (char *)COMPILER_RT_ALLOCA(Length + 1); + strncpy(Copy, Filename, Length + 1); + __llvm_profile_recursive_mkdir(Copy); + } +} + /* Open the profile data for merging. It opens the file in r+b mode with * file locking. If the file has content which is compatible with the * current process, it also reads in the profile data in the file and merge @@ -180,23 +208,23 @@ static int doProfileMerging(FILE *ProfileFile) { * dumper. With profile merging enabled, each executable as well as any of * its instrumented shared libraries dump profile data into their own data file. */ -static FILE *openFileForMerging(const char *ProfileFileName) { +static FILE *openFileForMerging(const char *ProfileFileName, int *MergeDone) { FILE *ProfileFile; int rc; + createProfileDir(ProfileFileName); ProfileFile = lprofOpenFileEx(ProfileFileName); if (!ProfileFile) return NULL; - rc = doProfileMerging(ProfileFile); - if (rc || COMPILER_RT_FTRUNCATE(ProfileFile, 0L) || + rc = doProfileMerging(ProfileFile, MergeDone); + if (rc || (!*MergeDone && COMPILER_RT_FTRUNCATE(ProfileFile, 0L)) || fseek(ProfileFile, 0L, SEEK_SET) == -1) { PROF_ERR("Profile Merging of file %s failed: %s\n", ProfileFileName, strerror(errno)); fclose(ProfileFile); return NULL; } - fseek(ProfileFile, 0L, SEEK_SET); return ProfileFile; } @@ -205,17 +233,20 @@ static int writeFile(const char *OutputName) { int RetVal; FILE *OutputFile; + int MergeDone = 0; if (!doMerging()) OutputFile = fopen(OutputName, "ab"); else - OutputFile = openFileForMerging(OutputName); + OutputFile = openFileForMerging(OutputName, &MergeDone); if (!OutputFile) return -1; FreeHook = &free; setupIOBuffer(); - RetVal = lprofWriteData(fileWriter, OutputFile, lprofGetVPDataReader()); + ProfDataWriter fileWriter; + initFileWriter(&fileWriter, OutputFile); + RetVal = lprofWriteData(&fileWriter, lprofGetVPDataReader(), MergeDone); fclose(OutputFile); return RetVal; @@ -233,18 +264,13 @@ static void truncateCurrentFile(void) { if (!Filename) return; - /* Create the directory holding the file, if needed. */ - if (lprofFindFirstDirSeparator(Filename)) { - char *Copy = (char *)COMPILER_RT_ALLOCA(Length + 1); - strncpy(Copy, Filename, Length + 1); - __llvm_profile_recursive_mkdir(Copy); - } - /* By pass file truncation to allow online raw profile * merging. */ if (lprofCurFilename.MergePoolSize) return; + createProfileDir(Filename); + /* Truncate the file. Later we'll reopen and append. */ File = fopen(Filename, "w"); if (!File) @@ -524,6 +550,7 @@ int __llvm_profile_write_file(void) { int rc, Length; const char *Filename; char *FilenameBuf; + int PDeathSig = 0; if (lprofProfileDumped()) { PROF_NOTE("Profile data not written to file: %s.\n", @@ -550,10 +577,18 @@ int __llvm_profile_write_file(void) { return -1; } + // Temporarily suspend getting SIGKILL when the parent exits. + PDeathSig = lprofSuspendSigKill(); + /* Write profile data to the file. */ rc = writeFile(Filename); if (rc) PROF_ERR("Failed to write file \"%s\": %s\n", Filename, strerror(errno)); + + // Restore SIGKILL. + if (PDeathSig == 1) + lprofRestoreSigKill(); + return rc; } diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingInternal.h b/contrib/compiler-rt/lib/profile/InstrProfilingInternal.h index c73b291..36490ef 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingInternal.h +++ b/contrib/compiler-rt/lib/profile/InstrProfilingInternal.h @@ -48,17 +48,21 @@ typedef struct ProfDataIOVec { size_t NumElm; } ProfDataIOVec; -typedef uint32_t (*WriterCallback)(ProfDataIOVec *, uint32_t NumIOVecs, - void **WriterCtx); +struct ProfDataWriter; +typedef uint32_t (*WriterCallback)(struct ProfDataWriter *This, ProfDataIOVec *, + uint32_t NumIOVecs); + +typedef struct ProfDataWriter { + WriterCallback Write; + void *WriterCtx; +} ProfDataWriter; /*! * The data structure for buffered IO of profile data. */ typedef struct ProfBufferIO { - /* File handle. */ - void *File; - /* Low level IO callback. */ - WriterCallback FileWriter; + ProfDataWriter *FileWriter; + uint32_t OwnFileWriter; /* The start of the buffer. */ uint8_t *BufferStart; /* Total size of the buffer. */ @@ -73,7 +77,7 @@ ProfBufferIO *lprofCreateBufferIOInternal(void *File, uint32_t BufferSz); /*! * This is the interface to create a handle for buffered IO. */ -ProfBufferIO *lprofCreateBufferIO(WriterCallback FileWriter, void *File); +ProfBufferIO *lprofCreateBufferIO(ProfDataWriter *FileWriter); /*! * The interface to destroy the bufferIO handle and reclaim @@ -96,8 +100,9 @@ int lprofBufferIOFlush(ProfBufferIO *BufferIO); /* The low level interface to write data into a buffer. It is used as the * callback by other high level writer methods such as buffered IO writer * and profile data writer. */ -uint32_t lprofBufferWriter(ProfDataIOVec *IOVecs, uint32_t NumIOVecs, - void **WriterCtx); +uint32_t lprofBufferWriter(ProfDataWriter *This, ProfDataIOVec *IOVecs, + uint32_t NumIOVecs); +void initBufferWriter(ProfDataWriter *BufferWriter, char *Buffer); struct ValueProfData; struct ValueProfRecord; @@ -133,15 +138,17 @@ typedef struct VPDataReaderType { uint32_t N); } VPDataReaderType; -int lprofWriteData(WriterCallback Writer, void *WriterCtx, - VPDataReaderType *VPDataReader); -int lprofWriteDataImpl(WriterCallback Writer, void *WriterCtx, +/* Write profile data to destinitation. If SkipNameDataWrite is set to 1, + the name data is already in destintation, we just skip over it. */ +int lprofWriteData(ProfDataWriter *Writer, VPDataReaderType *VPDataReader, + int SkipNameDataWrite); +int lprofWriteDataImpl(ProfDataWriter *Writer, const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd, const uint64_t *CountersBegin, const uint64_t *CountersEnd, VPDataReaderType *VPDataReader, const char *NamesBegin, - const char *NamesEnd); + const char *NamesEnd, int SkipNameDataWrite); /* Merge value profile data pointed to by SrcValueProfData into * in-memory profile counters pointed by to DstData. */ diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingNameVar.c b/contrib/compiler-rt/lib/profile/InstrProfilingNameVar.c new file mode 100644 index 0000000..264568f --- /dev/null +++ b/contrib/compiler-rt/lib/profile/InstrProfilingNameVar.c @@ -0,0 +1,18 @@ +/*===- InstrProfilingNameVar.c - profile name variable setup -------------===*\ +|* +|* The LLVM Compiler Infrastructure +|* +|* This file is distributed under the University of Illinois Open Source +|* License. See LICENSE.TXT for details. +|* +\*===----------------------------------------------------------------------===*/ + +#include "InstrProfiling.h" + +/* char __llvm_profile_filename[1] + * + * The runtime should only provide its own definition of this symbol when the + * user has not specified one. Set this up by moving the runtime's copy of this + * symbol to an object file within the archive. + */ +COMPILER_RT_WEAK char INSTR_PROF_PROFILE_NAME_VAR[1] = {0}; diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingUtil.c b/contrib/compiler-rt/lib/profile/InstrProfilingUtil.c index 321c719..fb68f30 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingUtil.c +++ b/contrib/compiler-rt/lib/profile/InstrProfilingUtil.c @@ -29,6 +29,11 @@ #include <stdlib.h> #include <string.h> +#if defined(__linux__) +#include <signal.h> +#include <sys/prctl.h> +#endif + COMPILER_RT_VISIBILITY void __llvm_profile_recursive_mkdir(char *path) { int i; @@ -219,3 +224,21 @@ COMPILER_RT_VISIBILITY const char *lprofFindLastDirSeparator(const char *Path) { #endif return Sep; } + +COMPILER_RT_VISIBILITY int lprofSuspendSigKill() { +#if defined(__linux__) + int PDeachSig = 0; + /* Temporarily suspend getting SIGKILL upon exit of the parent process. */ + if (prctl(PR_GET_PDEATHSIG, &PDeachSig) == 0 && PDeachSig == SIGKILL) + prctl(PR_SET_PDEATHSIG, 0); + return (PDeachSig == SIGKILL); +#else + return 0; +#endif +} + +COMPILER_RT_VISIBILITY void lprofRestoreSigKill() { +#if defined(__linux__) + prctl(PR_SET_PDEATHSIG, SIGKILL); +#endif +} diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingUtil.h b/contrib/compiler-rt/lib/profile/InstrProfilingUtil.h index a80fde7..9698599 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingUtil.h +++ b/contrib/compiler-rt/lib/profile/InstrProfilingUtil.h @@ -51,4 +51,12 @@ int lprofGetHostName(char *Name, int Len); unsigned lprofBoolCmpXchg(void **Ptr, void *OldV, void *NewV); void *lprofPtrFetchAdd(void **Mem, long ByteIncr); +/* Temporarily suspend SIGKILL. Return value of 1 means a restore is needed. + * Other return values mean no restore is needed. + */ +int lprofSuspendSigKill(); + +/* Restore previously suspended SIGKILL. */ +void lprofRestoreSigKill(); + #endif /* PROFILE_INSTRPROFILINGUTIL_H */ diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingValue.c b/contrib/compiler-rt/lib/profile/InstrProfilingValue.c index 6648f89..44263da 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingValue.c +++ b/contrib/compiler-rt/lib/profile/InstrProfilingValue.c @@ -220,6 +220,35 @@ __llvm_profile_instrument_target(uint64_t TargetValue, void *Data, } /* + * The target values are partitioned into multiple regions/ranges. There is one + * contiguous region which is precise -- every value in the range is tracked + * individually. A value outside the precise region will be collapsed into one + * value depending on the region it falls in. + * + * There are three regions: + * 1. (-inf, PreciseRangeStart) and (PreciseRangeLast, LargeRangeValue) belong + * to one region -- all values here should be mapped to one value of + * "PreciseRangeLast + 1". + * 2. [PreciseRangeStart, PreciseRangeLast] + * 3. Large values: [LargeValue, +inf) maps to one value of LargeValue. + * + * The range for large values is optional. The default value of INT64_MIN + * indicates it is not specified. + */ +COMPILER_RT_VISIBILITY void __llvm_profile_instrument_range( + uint64_t TargetValue, void *Data, uint32_t CounterIndex, + int64_t PreciseRangeStart, int64_t PreciseRangeLast, int64_t LargeValue) { + + if (LargeValue != INT64_MIN && (int64_t)TargetValue >= LargeValue) + TargetValue = LargeValue; + else if ((int64_t)TargetValue < PreciseRangeStart || + (int64_t)TargetValue > PreciseRangeLast) + TargetValue = PreciseRangeLast + 1; + + __llvm_profile_instrument_target(TargetValue, Data, CounterIndex); +} + +/* * A wrapper struct that represents value profile runtime data. * Like InstrProfRecord class which is used by profiling host tools, * ValueProfRuntimeRecord also implements the abstract intefaces defined in diff --git a/contrib/compiler-rt/lib/profile/InstrProfilingWriter.c b/contrib/compiler-rt/lib/profile/InstrProfilingWriter.c index 95f37e8..d4c9b9b 100644 --- a/contrib/compiler-rt/lib/profile/InstrProfilingWriter.c +++ b/contrib/compiler-rt/lib/profile/InstrProfilingWriter.c @@ -31,41 +31,44 @@ COMPILER_RT_VISIBILITY uint32_t VPBufferSize = 0; /* The buffer writer is reponsponsible in keeping writer state * across the call. */ -COMPILER_RT_VISIBILITY uint32_t lprofBufferWriter(ProfDataIOVec *IOVecs, - uint32_t NumIOVecs, - void **WriterCtx) { +COMPILER_RT_VISIBILITY uint32_t lprofBufferWriter(ProfDataWriter *This, + ProfDataIOVec *IOVecs, + uint32_t NumIOVecs) { uint32_t I; - char **Buffer = (char **)WriterCtx; + char **Buffer = (char **)&This->WriterCtx; for (I = 0; I < NumIOVecs; I++) { size_t Length = IOVecs[I].ElmSize * IOVecs[I].NumElm; - memcpy(*Buffer, IOVecs[I].Data, Length); + if (IOVecs[I].Data) + memcpy(*Buffer, IOVecs[I].Data, Length); *Buffer += Length; } return 0; } -static void llvmInitBufferIO(ProfBufferIO *BufferIO, WriterCallback FileWriter, - void *File, uint8_t *Buffer, uint32_t BufferSz) { - BufferIO->File = File; +static void llvmInitBufferIO(ProfBufferIO *BufferIO, ProfDataWriter *FileWriter, + uint8_t *Buffer, uint32_t BufferSz) { BufferIO->FileWriter = FileWriter; + BufferIO->OwnFileWriter = 0; BufferIO->BufferStart = Buffer; BufferIO->BufferSz = BufferSz; BufferIO->CurOffset = 0; } COMPILER_RT_VISIBILITY ProfBufferIO * -lprofCreateBufferIO(WriterCallback FileWriter, void *File) { +lprofCreateBufferIO(ProfDataWriter *FileWriter) { uint8_t *Buffer = DynamicBufferIOBuffer; uint32_t BufferSize = VPBufferSize; if (!Buffer) { Buffer = &BufferIOBuffer[0]; BufferSize = sizeof(BufferIOBuffer); } - llvmInitBufferIO(&TheBufferIO, FileWriter, File, Buffer, BufferSize); + llvmInitBufferIO(&TheBufferIO, FileWriter, Buffer, BufferSize); return &TheBufferIO; } COMPILER_RT_VISIBILITY void lprofDeleteBufferIO(ProfBufferIO *BufferIO) { + if (BufferIO->OwnFileWriter) + FreeHook(BufferIO->FileWriter); if (DynamicBufferIOBuffer) { FreeHook(DynamicBufferIOBuffer); DynamicBufferIOBuffer = 0; @@ -83,13 +86,16 @@ lprofBufferIOWrite(ProfBufferIO *BufferIO, const uint8_t *Data, uint32_t Size) { /* Special case, bypass the buffer completely. */ ProfDataIOVec IO[] = {{Data, sizeof(uint8_t), Size}}; if (Size > BufferIO->BufferSz) { - if (BufferIO->FileWriter(IO, 1, &BufferIO->File)) + if (BufferIO->FileWriter->Write(BufferIO->FileWriter, IO, 1)) return -1; } else { /* Write the data to buffer */ uint8_t *Buffer = BufferIO->BufferStart + BufferIO->CurOffset; - lprofBufferWriter(IO, 1, (void **)&Buffer); - BufferIO->CurOffset = Buffer - BufferIO->BufferStart; + ProfDataWriter BufferWriter; + initBufferWriter(&BufferWriter, (char *)Buffer); + lprofBufferWriter(&BufferWriter, IO, 1); + BufferIO->CurOffset = + (uint8_t *)BufferWriter.WriterCtx - BufferIO->BufferStart; } return 0; } @@ -98,7 +104,7 @@ COMPILER_RT_VISIBILITY int lprofBufferIOFlush(ProfBufferIO *BufferIO) { if (BufferIO->CurOffset) { ProfDataIOVec IO[] = { {BufferIO->BufferStart, sizeof(uint8_t), BufferIO->CurOffset}}; - if (BufferIO->FileWriter(IO, 1, &BufferIO->File)) + if (BufferIO->FileWriter->Write(BufferIO->FileWriter, IO, 1)) return -1; BufferIO->CurOffset = 0; } @@ -201,7 +207,7 @@ static int writeOneValueProfData(ProfBufferIO *BufferIO, return 0; } -static int writeValueProfData(WriterCallback Writer, void *WriterCtx, +static int writeValueProfData(ProfDataWriter *Writer, VPDataReaderType *VPDataReader, const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd) { @@ -211,7 +217,7 @@ static int writeValueProfData(WriterCallback Writer, void *WriterCtx, if (!VPDataReader) return 0; - BufferIO = lprofCreateBufferIO(Writer, WriterCtx); + BufferIO = lprofCreateBufferIO(Writer); for (DI = DataBegin; DI < DataEnd; DI++) { if (writeOneValueProfData(BufferIO, VPDataReader, DI)) @@ -225,9 +231,9 @@ static int writeValueProfData(WriterCallback Writer, void *WriterCtx, return 0; } -COMPILER_RT_VISIBILITY int lprofWriteData(WriterCallback Writer, - void *WriterCtx, - VPDataReaderType *VPDataReader) { +COMPILER_RT_VISIBILITY int lprofWriteData(ProfDataWriter *Writer, + VPDataReaderType *VPDataReader, + int SkipNameDataWrite) { /* Match logic in __llvm_profile_write_buffer(). */ const __llvm_profile_data *DataBegin = __llvm_profile_begin_data(); const __llvm_profile_data *DataEnd = __llvm_profile_end_data(); @@ -235,18 +241,17 @@ COMPILER_RT_VISIBILITY int lprofWriteData(WriterCallback Writer, const uint64_t *CountersEnd = __llvm_profile_end_counters(); const char *NamesBegin = __llvm_profile_begin_names(); const char *NamesEnd = __llvm_profile_end_names(); - return lprofWriteDataImpl(Writer, WriterCtx, DataBegin, DataEnd, - CountersBegin, CountersEnd, VPDataReader, - NamesBegin, NamesEnd); + return lprofWriteDataImpl(Writer, DataBegin, DataEnd, CountersBegin, + CountersEnd, VPDataReader, NamesBegin, NamesEnd, + SkipNameDataWrite); } COMPILER_RT_VISIBILITY int -lprofWriteDataImpl(WriterCallback Writer, void *WriterCtx, - const __llvm_profile_data *DataBegin, +lprofWriteDataImpl(ProfDataWriter *Writer, const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd, const uint64_t *CountersBegin, const uint64_t *CountersEnd, VPDataReaderType *VPDataReader, const char *NamesBegin, - const char *NamesEnd) { + const char *NamesEnd, int SkipNameDataWrite) { /* Calculate size of sections. */ const uint64_t DataSize = __llvm_profile_get_data_size(DataBegin, DataEnd); @@ -268,14 +273,14 @@ lprofWriteDataImpl(WriterCallback Writer, void *WriterCtx, #include "InstrProfData.inc" /* Write the data. */ - ProfDataIOVec IOVec[] = {{&Header, sizeof(__llvm_profile_header), 1}, - {DataBegin, sizeof(__llvm_profile_data), DataSize}, - {CountersBegin, sizeof(uint64_t), CountersSize}, - {NamesBegin, sizeof(uint8_t), NamesSize}, - {Zeroes, sizeof(uint8_t), Padding}}; - if (Writer(IOVec, sizeof(IOVec) / sizeof(*IOVec), &WriterCtx)) + ProfDataIOVec IOVec[] = { + {&Header, sizeof(__llvm_profile_header), 1}, + {DataBegin, sizeof(__llvm_profile_data), DataSize}, + {CountersBegin, sizeof(uint64_t), CountersSize}, + {SkipNameDataWrite ? NULL : NamesBegin, sizeof(uint8_t), NamesSize}, + {Zeroes, sizeof(uint8_t), Padding}}; + if (Writer->Write(Writer, IOVec, sizeof(IOVec) / sizeof(*IOVec))) return -1; - return writeValueProfData(Writer, WriterCtx, VPDataReader, DataBegin, - DataEnd); + return writeValueProfData(Writer, VPDataReader, DataBegin, DataEnd); } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.cc b/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.cc index 08fd2a4..9abb5b5 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.cc @@ -15,10 +15,9 @@ #include "sanitizer_flag_parser.h" #include "sanitizer_platform.h" -#if !SANITIZER_LINUX -// other platforms do not have weak symbols out of the box. -extern "C" const char* __sancov_default_options() { return ""; } -#endif +SANITIZER_INTERFACE_WEAK_DEF(const char*, __sancov_default_options, void) { + return ""; +} using namespace __sanitizer; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.h b/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.h index 5fbd7ad..627d9a3 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sancov_flags.h @@ -32,9 +32,9 @@ inline SancovFlags* sancov_flags() { return &sancov_flags_dont_use_directly; } void InitializeSancovFlags(); +} // namespace __sancov + extern "C" SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE const char* __sancov_default_options(); -} // namespace __sancov - #endif diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc index d47b5b4..84f523c 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc @@ -14,6 +14,7 @@ #include "sanitizer_allocator.h" +#include "sanitizer_allocator_checks.h" #include "sanitizer_allocator_internal.h" #include "sanitizer_atomic.h" #include "sanitizer_common.h" @@ -94,8 +95,7 @@ InternalAllocator *internal_allocator() { SpinMutexLock l(&internal_alloc_init_mu); if (atomic_load(&internal_allocator_initialized, memory_order_relaxed) == 0) { - internal_allocator_instance->Init( - /* may_return_null */ false, kReleaseToOSIntervalNever); + internal_allocator_instance->Init(kReleaseToOSIntervalNever); atomic_store(&internal_allocator_initialized, 1, memory_order_release); } } @@ -108,9 +108,9 @@ static void *RawInternalAlloc(uptr size, InternalAllocatorCache *cache, if (cache == 0) { SpinMutexLock l(&internal_allocator_cache_mu); return internal_allocator()->Allocate(&internal_allocator_cache, size, - alignment, false); + alignment); } - return internal_allocator()->Allocate(cache, size, alignment, false); + return internal_allocator()->Allocate(cache, size, alignment); } static void *RawInternalRealloc(void *ptr, uptr size, @@ -161,8 +161,8 @@ void *InternalRealloc(void *addr, uptr size, InternalAllocatorCache *cache) { } void *InternalCalloc(uptr count, uptr size, InternalAllocatorCache *cache) { - if (CallocShouldReturnNullDueToOverflow(count, size)) - return internal_allocator()->ReturnNullOrDieOnBadRequest(); + if (UNLIKELY(CheckForCallocOverflow(count, size))) + return InternalAllocator::FailureHandler::OnBadRequest(); void *p = InternalAlloc(count * size, cache); if (p) internal_memset(p, 0, count * size); return p; @@ -203,18 +203,15 @@ void SetLowLevelAllocateCallback(LowLevelAllocateCallback callback) { low_level_alloc_callback = callback; } -bool CallocShouldReturnNullDueToOverflow(uptr size, uptr n) { - if (!size) return false; - uptr max = (uptr)-1L; - return (max / size) < n; -} - -static atomic_uint8_t reporting_out_of_memory = {0}; +static atomic_uint8_t allocator_out_of_memory = {0}; +static atomic_uint8_t allocator_may_return_null = {0}; -bool IsReportingOOM() { return atomic_load_relaxed(&reporting_out_of_memory); } +bool IsAllocatorOutOfMemory() { + return atomic_load_relaxed(&allocator_out_of_memory); +} -void NORETURN ReportAllocatorCannotReturnNull(bool out_of_memory) { - if (out_of_memory) atomic_store_relaxed(&reporting_out_of_memory, 1); +// Prints error message and kills the program. +void NORETURN ReportAllocatorCannotReturnNull() { Report("%s's allocator is terminating the process instead of returning 0\n", SanitizerToolName); Report("If you don't like this behavior set allocator_may_return_null=1\n"); @@ -222,4 +219,35 @@ void NORETURN ReportAllocatorCannotReturnNull(bool out_of_memory) { Die(); } +bool AllocatorMayReturnNull() { + return atomic_load(&allocator_may_return_null, memory_order_relaxed); +} + +void SetAllocatorMayReturnNull(bool may_return_null) { + atomic_store(&allocator_may_return_null, may_return_null, + memory_order_relaxed); +} + +void *ReturnNullOrDieOnFailure::OnBadRequest() { + if (AllocatorMayReturnNull()) + return nullptr; + ReportAllocatorCannotReturnNull(); +} + +void *ReturnNullOrDieOnFailure::OnOOM() { + atomic_store_relaxed(&allocator_out_of_memory, 1); + if (AllocatorMayReturnNull()) + return nullptr; + ReportAllocatorCannotReturnNull(); +} + +void NORETURN *DieOnFailure::OnBadRequest() { + ReportAllocatorCannotReturnNull(); +} + +void NORETURN *DieOnFailure::OnOOM() { + atomic_store_relaxed(&allocator_out_of_memory, 1); + ReportAllocatorCannotReturnNull(); +} + } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h index 9a37a2f..8c5696e 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h @@ -24,12 +24,28 @@ namespace __sanitizer { -// Returns true if ReportAllocatorCannotReturnNull(true) was called. -// Can be use to avoid memory hungry operations. -bool IsReportingOOM(); +// Since flags are immutable and allocator behavior can be changed at runtime +// (unit tests or ASan on Android are some examples), allocator_may_return_null +// flag value is cached here and can be altered later. +bool AllocatorMayReturnNull(); +void SetAllocatorMayReturnNull(bool may_return_null); -// Prints error message and kills the program. -void NORETURN ReportAllocatorCannotReturnNull(bool out_of_memory); +// Allocator failure handling policies: +// Implements AllocatorMayReturnNull policy, returns null when the flag is set, +// dies otherwise. +struct ReturnNullOrDieOnFailure { + static void *OnBadRequest(); + static void *OnOOM(); +}; +// Always dies on the failure. +struct DieOnFailure { + static void NORETURN *OnBadRequest(); + static void NORETURN *OnOOM(); +}; + +// Returns true if allocator detected OOM condition. Can be used to avoid memory +// hungry operations. Set when AllocatorReturnNullOrDieOnOOM() is called. +bool IsAllocatorOutOfMemory(); // Allocators call these callbacks on mmap/munmap. struct NoOpMapUnmapCallback { @@ -40,9 +56,6 @@ struct NoOpMapUnmapCallback { // Callback type for iterating over chunks. typedef void (*ForEachChunkCallback)(uptr chunk, void *arg); -// Returns true if calloc(size, n) should return 0 due to overflow in size*n. -bool CallocShouldReturnNullDueToOverflow(uptr size, uptr n); - #include "sanitizer_allocator_size_class_map.h" #include "sanitizer_allocator_stats.h" #include "sanitizer_allocator_primary64.h" diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h new file mode 100644 index 0000000..202916e --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h @@ -0,0 +1,64 @@ +//===-- sanitizer_allocator_checks.h ----------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// Various checks shared between ThreadSanitizer, MemorySanitizer, etc. memory +// allocators. +// +//===----------------------------------------------------------------------===// + +#ifndef SANITIZER_ALLOCATOR_CHECKS_H +#define SANITIZER_ALLOCATOR_CHECKS_H + +#include "sanitizer_errno.h" +#include "sanitizer_internal_defs.h" +#include "sanitizer_common.h" +#include "sanitizer_platform.h" + +namespace __sanitizer { + +// A common errno setting logic shared by almost all sanitizer allocator APIs. +INLINE void *SetErrnoOnNull(void *ptr) { + if (UNLIKELY(!ptr)) + errno = errno_ENOMEM; + return ptr; +} + +// In case of the check failure, the caller of the following Check... functions +// should "return POLICY::OnBadRequest();" where POLICY is the current allocator +// failure handling policy. + +// Checks aligned_alloc() parameters, verifies that the alignment is a power of +// two and that the size is a multiple of alignment for POSIX implementation, +// and a bit relaxed requirement for non-POSIX ones, that the size is a multiple +// of alignment. +INLINE bool CheckAlignedAllocAlignmentAndSize(uptr alignment, uptr size) { +#if SANITIZER_POSIX + return IsPowerOfTwo(alignment) && (size & (alignment - 1)) == 0; +#else + return size % alignment == 0; +#endif +} + +// Checks posix_memalign() parameters, verifies that alignment is a power of two +// and a multiple of sizeof(void *). +INLINE bool CheckPosixMemalignAlignment(uptr alignment) { + return IsPowerOfTwo(alignment) && (alignment % sizeof(void *)) == 0; // NOLINT +} + +// Returns true if calloc(size, n) call overflows on size*n calculation. +INLINE bool CheckForCallocOverflow(uptr size, uptr n) { + if (!size) + return false; + uptr max = (uptr)-1L; + return (max / size) < n; +} + +} // namespace __sanitizer + +#endif // SANITIZER_ALLOCATOR_CHECKS_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_combined.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_combined.h index 19e1ae9..efd25ca 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_combined.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_combined.h @@ -24,31 +24,26 @@ template <class PrimaryAllocator, class AllocatorCache, class SecondaryAllocator> // NOLINT class CombinedAllocator { public: - void InitCommon(bool may_return_null, s32 release_to_os_interval_ms) { - primary_.Init(release_to_os_interval_ms); - atomic_store(&may_return_null_, may_return_null, memory_order_relaxed); - } + typedef typename SecondaryAllocator::FailureHandler FailureHandler; - void InitLinkerInitialized( - bool may_return_null, s32 release_to_os_interval_ms) { - secondary_.InitLinkerInitialized(may_return_null); + void InitLinkerInitialized(s32 release_to_os_interval_ms) { + primary_.Init(release_to_os_interval_ms); + secondary_.InitLinkerInitialized(); stats_.InitLinkerInitialized(); - InitCommon(may_return_null, release_to_os_interval_ms); } - void Init(bool may_return_null, s32 release_to_os_interval_ms) { - secondary_.Init(may_return_null); + void Init(s32 release_to_os_interval_ms) { + primary_.Init(release_to_os_interval_ms); + secondary_.Init(); stats_.Init(); - InitCommon(may_return_null, release_to_os_interval_ms); } - void *Allocate(AllocatorCache *cache, uptr size, uptr alignment, - bool cleared = false, bool check_rss_limit = false) { + void *Allocate(AllocatorCache *cache, uptr size, uptr alignment) { // Returning 0 on malloc(0) may break a lot of code. if (size == 0) size = 1; - if (size + alignment < size) return ReturnNullOrDieOnBadRequest(); - if (check_rss_limit && RssLimitIsExceeded()) return ReturnNullOrDieOnOOM(); + if (size + alignment < size) + return FailureHandler::OnBadRequest(); uptr original_size = size; // If alignment requirements are to be fulfilled by the frontend allocator // rather than by the primary or secondary, passing an alignment lower than @@ -56,48 +51,24 @@ class CombinedAllocator { // alignment check. if (alignment > 8) size = RoundUpTo(size, alignment); - void *res; - bool from_primary = primary_.CanAllocate(size, alignment); // The primary allocator should return a 2^x aligned allocation when // requested 2^x bytes, hence using the rounded up 'size' when being // serviced by the primary (this is no longer true when the primary is // using a non-fixed base address). The secondary takes care of the // alignment without such requirement, and allocating 'size' would use // extraneous memory, so we employ 'original_size'. - if (from_primary) + void *res; + if (primary_.CanAllocate(size, alignment)) res = cache->Allocate(&primary_, primary_.ClassID(size)); else res = secondary_.Allocate(&stats_, original_size, alignment); + if (!res) + return FailureHandler::OnOOM(); if (alignment > 8) CHECK_EQ(reinterpret_cast<uptr>(res) & (alignment - 1), 0); - // When serviced by the secondary, the chunk comes from a mmap allocation - // and will be zero'd out anyway. We only need to clear our the chunk if - // it was serviced by the primary, hence using the rounded up 'size'. - if (cleared && res && from_primary) - internal_bzero_aligned16(res, RoundUpTo(size, 16)); return res; } - bool MayReturnNull() const { - return atomic_load(&may_return_null_, memory_order_acquire); - } - - void *ReturnNullOrDieOnBadRequest() { - if (MayReturnNull()) - return nullptr; - ReportAllocatorCannotReturnNull(false); - } - - void *ReturnNullOrDieOnOOM() { - if (MayReturnNull()) return nullptr; - ReportAllocatorCannotReturnNull(true); - } - - void SetMayReturnNull(bool may_return_null) { - secondary_.SetMayReturnNull(may_return_null); - atomic_store(&may_return_null_, may_return_null, memory_order_release); - } - s32 ReleaseToOSIntervalMs() const { return primary_.ReleaseToOSIntervalMs(); } @@ -106,15 +77,6 @@ class CombinedAllocator { primary_.SetReleaseToOSIntervalMs(release_to_os_interval_ms); } - bool RssLimitIsExceeded() { - return atomic_load(&rss_limit_is_exceeded_, memory_order_acquire); - } - - void SetRssLimitIsExceeded(bool rss_limit_is_exceeded) { - atomic_store(&rss_limit_is_exceeded_, rss_limit_is_exceeded, - memory_order_release); - } - void Deallocate(AllocatorCache *cache, void *p) { if (!p) return; if (primary_.PointerIsMine(p)) @@ -227,7 +189,5 @@ class CombinedAllocator { PrimaryAllocator primary_; SecondaryAllocator secondary_; AllocatorGlobalStats stats_; - atomic_uint8_t may_return_null_; - atomic_uint8_t rss_limit_is_exceeded_; }; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_interface.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_interface.h index 5ff6edb..13910e7 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_interface.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_interface.h @@ -34,13 +34,12 @@ SANITIZER_INTERFACE_ATTRIBUTE int __sanitizer_install_malloc_and_free_hooks( void (*free_hook)(const void *)); SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE - /* OPTIONAL */ void __sanitizer_malloc_hook(void *ptr, uptr size); + void __sanitizer_malloc_hook(void *ptr, uptr size); SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE - /* OPTIONAL */ void __sanitizer_free_hook(void *ptr); + void __sanitizer_free_hook(void *ptr); - -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE - void __sanitizer_print_memory_profile(int top_percent); +SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void +__sanitizer_print_memory_profile(uptr top_percent, uptr max_number_of_contexts); } // extern "C" #endif // SANITIZER_ALLOCATOR_INTERFACE_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_internal.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_internal.h index e939cbe..a791d0d 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_internal.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_internal.h @@ -23,27 +23,32 @@ namespace __sanitizer { // purposes. typedef CompactSizeClassMap InternalSizeClassMap; -static const uptr kInternalAllocatorSpace = 0; -static const u64 kInternalAllocatorSize = SANITIZER_MMAP_RANGE_SIZE; static const uptr kInternalAllocatorRegionSizeLog = 20; -#if SANITIZER_WORDSIZE == 32 static const uptr kInternalAllocatorNumRegions = - kInternalAllocatorSize >> kInternalAllocatorRegionSizeLog; + SANITIZER_MMAP_RANGE_SIZE >> kInternalAllocatorRegionSizeLog; +#if SANITIZER_WORDSIZE == 32 typedef FlatByteMap<kInternalAllocatorNumRegions> ByteMap; #else -static const uptr kInternalAllocatorNumRegions = - kInternalAllocatorSize >> kInternalAllocatorRegionSizeLog; typedef TwoLevelByteMap<(kInternalAllocatorNumRegions >> 12), 1 << 12> ByteMap; #endif -typedef SizeClassAllocator32< - kInternalAllocatorSpace, kInternalAllocatorSize, 0, InternalSizeClassMap, - kInternalAllocatorRegionSizeLog, ByteMap> PrimaryInternalAllocator; +struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = 0; + typedef InternalSizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = kInternalAllocatorRegionSizeLog; + typedef __sanitizer::ByteMap ByteMap; + typedef NoOpMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; +}; +typedef SizeClassAllocator32<AP32> PrimaryInternalAllocator; typedef SizeClassAllocatorLocalCache<PrimaryInternalAllocator> InternalAllocatorCache; typedef CombinedAllocator<PrimaryInternalAllocator, InternalAllocatorCache, - LargeMmapAllocator<> > InternalAllocator; + LargeMmapAllocator<NoOpMapUnmapCallback, DieOnFailure> + > InternalAllocator; void *InternalAlloc(uptr size, InternalAllocatorCache *cache = nullptr, uptr alignment = 0); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h index e1172e0..ec0742c 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h @@ -45,10 +45,12 @@ struct SizeClassAllocator64LocalCache { void *Allocate(SizeClassAllocator *allocator, uptr class_id) { CHECK_NE(class_id, 0UL); CHECK_LT(class_id, kNumClasses); - stats_.Add(AllocatorStatAllocated, Allocator::ClassIdToSize(class_id)); PerClass *c = &per_class_[class_id]; - if (UNLIKELY(c->count == 0)) - Refill(c, allocator, class_id); + if (UNLIKELY(c->count == 0)) { + if (UNLIKELY(!Refill(c, allocator, class_id))) + return nullptr; + } + stats_.Add(AllocatorStatAllocated, c->class_size); CHECK_GT(c->count, 0); CompactPtrT chunk = c->chunks[--c->count]; void *res = reinterpret_cast<void *>(allocator->CompactPtrToPointer( @@ -62,8 +64,8 @@ struct SizeClassAllocator64LocalCache { // If the first allocator call on a new thread is a deallocation, then // max_count will be zero, leading to check failure. InitCache(); - stats_.Sub(AllocatorStatAllocated, Allocator::ClassIdToSize(class_id)); PerClass *c = &per_class_[class_id]; + stats_.Sub(AllocatorStatAllocated, c->class_size); CHECK_NE(c->max_count, 0UL); if (UNLIKELY(c->count == c->max_count)) Drain(c, allocator, class_id, c->max_count / 2); @@ -85,6 +87,7 @@ struct SizeClassAllocator64LocalCache { struct PerClass { u32 count; u32 max_count; + uptr class_size; CompactPtrT chunks[2 * SizeClassMap::kMaxNumCachedHint]; }; PerClass per_class_[kNumClasses]; @@ -96,16 +99,19 @@ struct SizeClassAllocator64LocalCache { for (uptr i = 0; i < kNumClasses; i++) { PerClass *c = &per_class_[i]; c->max_count = 2 * SizeClassMap::MaxCachedHint(i); + c->class_size = Allocator::ClassIdToSize(i); } } - NOINLINE void Refill(PerClass *c, SizeClassAllocator *allocator, + NOINLINE bool Refill(PerClass *c, SizeClassAllocator *allocator, uptr class_id) { InitCache(); - uptr num_requested_chunks = SizeClassMap::MaxCachedHint(class_id); - allocator->GetFromAllocator(&stats_, class_id, c->chunks, - num_requested_chunks); + uptr num_requested_chunks = c->max_count / 2; + if (UNLIKELY(!allocator->GetFromAllocator(&stats_, class_id, c->chunks, + num_requested_chunks))) + return false; c->count = num_requested_chunks; + return true; } NOINLINE void Drain(PerClass *c, SizeClassAllocator *allocator, uptr class_id, @@ -141,10 +147,12 @@ struct SizeClassAllocator32LocalCache { void *Allocate(SizeClassAllocator *allocator, uptr class_id) { CHECK_NE(class_id, 0UL); CHECK_LT(class_id, kNumClasses); - stats_.Add(AllocatorStatAllocated, Allocator::ClassIdToSize(class_id)); PerClass *c = &per_class_[class_id]; - if (UNLIKELY(c->count == 0)) - Refill(allocator, class_id); + if (UNLIKELY(c->count == 0)) { + if (UNLIKELY(!Refill(allocator, class_id))) + return nullptr; + } + stats_.Add(AllocatorStatAllocated, c->class_size); void *res = c->batch[--c->count]; PREFETCH(c->batch[c->count - 1]); return res; @@ -156,8 +164,8 @@ struct SizeClassAllocator32LocalCache { // If the first allocator call on a new thread is a deallocation, then // max_count will be zero, leading to check failure. InitCache(); - stats_.Sub(AllocatorStatAllocated, Allocator::ClassIdToSize(class_id)); PerClass *c = &per_class_[class_id]; + stats_.Sub(AllocatorStatAllocated, c->class_size); CHECK_NE(c->max_count, 0UL); if (UNLIKELY(c->count == c->max_count)) Drain(allocator, class_id); @@ -177,6 +185,8 @@ struct SizeClassAllocator32LocalCache { struct PerClass { uptr count; uptr max_count; + uptr class_size; + uptr class_id_for_transfer_batch; void *batch[2 * TransferBatch::kMaxNumCached]; }; PerClass per_class_[kNumClasses]; @@ -185,31 +195,31 @@ struct SizeClassAllocator32LocalCache { void InitCache() { if (per_class_[1].max_count) return; + // TransferBatch class is declared in SizeClassAllocator. + uptr class_id_for_transfer_batch = + SizeClassMap::ClassID(sizeof(TransferBatch)); for (uptr i = 0; i < kNumClasses; i++) { PerClass *c = &per_class_[i]; - c->max_count = 2 * TransferBatch::MaxCached(i); + uptr max_cached = TransferBatch::MaxCached(i); + c->max_count = 2 * max_cached; + c->class_size = Allocator::ClassIdToSize(i); + // We transfer chunks between central and thread-local free lists in + // batches. For small size classes we allocate batches separately. For + // large size classes we may use one of the chunks to store the batch. + // sizeof(TransferBatch) must be a power of 2 for more efficient + // allocation. + c->class_id_for_transfer_batch = (c->class_size < + TransferBatch::AllocationSizeRequiredForNElements(max_cached)) ? + class_id_for_transfer_batch : 0; } } - // TransferBatch class is declared in SizeClassAllocator. - // We transfer chunks between central and thread-local free lists in batches. - // For small size classes we allocate batches separately. - // For large size classes we may use one of the chunks to store the batch. - // sizeof(TransferBatch) must be a power of 2 for more efficient allocation. - static uptr SizeClassForTransferBatch(uptr class_id) { - if (Allocator::ClassIdToSize(class_id) < - TransferBatch::AllocationSizeRequiredForNElements( - TransferBatch::MaxCached(class_id))) - return SizeClassMap::ClassID(sizeof(TransferBatch)); - return 0; - } - // Returns a TransferBatch suitable for class_id. // For small size classes allocates the batch from the allocator. // For large size classes simply returns b. TransferBatch *CreateBatch(uptr class_id, SizeClassAllocator *allocator, TransferBatch *b) { - if (uptr batch_class_id = SizeClassForTransferBatch(class_id)) + if (uptr batch_class_id = per_class_[class_id].class_id_for_transfer_batch) return (TransferBatch*)Allocate(allocator, batch_class_id); return b; } @@ -219,18 +229,21 @@ struct SizeClassAllocator32LocalCache { // Does notthing for large size classes. void DestroyBatch(uptr class_id, SizeClassAllocator *allocator, TransferBatch *b) { - if (uptr batch_class_id = SizeClassForTransferBatch(class_id)) + if (uptr batch_class_id = per_class_[class_id].class_id_for_transfer_batch) Deallocate(allocator, batch_class_id, b); } - NOINLINE void Refill(SizeClassAllocator *allocator, uptr class_id) { + NOINLINE bool Refill(SizeClassAllocator *allocator, uptr class_id) { InitCache(); PerClass *c = &per_class_[class_id]; TransferBatch *b = allocator->AllocateBatch(&stats_, this, class_id); + if (UNLIKELY(!b)) + return false; CHECK_GT(b->Count(), 0); b->CopyToArray(c->batch); c->count = b->Count(); DestroyBatch(class_id, allocator, b); + return true; } NOINLINE void Drain(SizeClassAllocator *allocator, uptr class_id) { @@ -240,6 +253,10 @@ struct SizeClassAllocator32LocalCache { uptr first_idx_to_drain = c->count - cnt; TransferBatch *b = CreateBatch( class_id, allocator, (TransferBatch *)c->batch[first_idx_to_drain]); + // Failure to allocate a batch while releasing memory is non recoverable. + // TODO(alekseys): Figure out how to do it without allocating a new batch. + if (UNLIKELY(!b)) + DieOnFailure::OnOOM(); b->SetFromArray(allocator->GetRegionBeginBySizeClass(class_id), &c->batch[first_idx_to_drain], cnt); c->count -= cnt; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h index 2882afd..e858215 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h @@ -24,7 +24,8 @@ template<class SizeClassAllocator> struct SizeClassAllocator32LocalCache; // be returned by MmapOrDie(). // // Region: -// a result of a single call to MmapAlignedOrDie(kRegionSize, kRegionSize). +// a result of a single call to MmapAlignedOrDieOnFatalError(kRegionSize, +// kRegionSize). // Since the regions are aligned by kRegionSize, there are exactly // kNumPossibleRegions possible regions in the address space and so we keep // a ByteMap possible_regions to store the size classes of each Region. @@ -36,13 +37,27 @@ template<class SizeClassAllocator> struct SizeClassAllocator32LocalCache; // // In order to avoid false sharing the objects of this class should be // chache-line aligned. -template <const uptr kSpaceBeg, const u64 kSpaceSize, - const uptr kMetadataSize, class SizeClassMap, - const uptr kRegionSizeLog, - class ByteMap, - class MapUnmapCallback = NoOpMapUnmapCallback> + +struct SizeClassAllocator32FlagMasks { // Bit masks. + enum { + kRandomShuffleChunks = 1, + }; +}; + +template <class Params> class SizeClassAllocator32 { public: + static const uptr kSpaceBeg = Params::kSpaceBeg; + static const u64 kSpaceSize = Params::kSpaceSize; + static const uptr kMetadataSize = Params::kMetadataSize; + typedef typename Params::SizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = Params::kRegionSizeLog; + typedef typename Params::ByteMap ByteMap; + typedef typename Params::MapUnmapCallback MapUnmapCallback; + + static const bool kRandomShuffleChunks = + Params::kFlags & SizeClassAllocator32FlagMasks::kRandomShuffleChunks; + struct TransferBatch { static const uptr kMaxNumCached = SizeClassMap::kMaxNumCachedHint - 2; void SetFromArray(uptr region_beg_unused, void *batch[], uptr count) { @@ -86,8 +101,7 @@ class SizeClassAllocator32 { return SizeClassMap::Size(class_id); } - typedef SizeClassAllocator32<kSpaceBeg, kSpaceSize, kMetadataSize, - SizeClassMap, kRegionSizeLog, ByteMap, MapUnmapCallback> ThisT; + typedef SizeClassAllocator32<Params> ThisT; typedef SizeClassAllocator32LocalCache<ThisT> AllocatorCache; void Init(s32 release_to_os_interval_ms) { @@ -104,7 +118,6 @@ class SizeClassAllocator32 { } void *MapWithCallback(uptr size) { - size = RoundUpTo(size, GetPageSizeCached()); void *res = MmapOrDie(size, "SizeClassAllocator32"); MapUnmapCallback().OnMap((uptr)res, size); return res; @@ -136,8 +149,9 @@ class SizeClassAllocator32 { CHECK_LT(class_id, kNumClasses); SizeClassInfo *sci = GetSizeClassInfo(class_id); SpinMutexLock l(&sci->mutex); - if (sci->free_list.empty()) - PopulateFreeList(stat, c, sci, class_id); + if (sci->free_list.empty() && + UNLIKELY(!PopulateFreeList(stat, c, sci, class_id))) + return nullptr; CHECK(!sci->free_list.empty()); TransferBatch *b = sci->free_list.front(); sci->free_list.pop_front(); @@ -264,11 +278,13 @@ class SizeClassAllocator32 { uptr AllocateRegion(AllocatorStats *stat, uptr class_id) { CHECK_LT(class_id, kNumClasses); - uptr res = reinterpret_cast<uptr>(MmapAlignedOrDie(kRegionSize, kRegionSize, - "SizeClassAllocator32")); + uptr res = reinterpret_cast<uptr>(MmapAlignedOrDieOnFatalError( + kRegionSize, kRegionSize, "SizeClassAllocator32")); + if (UNLIKELY(!res)) + return 0; MapUnmapCallback().OnMap(res, kRegionSize); stat->Add(AllocatorStatMapped, kRegionSize); - CHECK_EQ(0U, (res & (kRegionSize - 1))); + CHECK(IsAligned(res, kRegionSize)); possible_regions.set(ComputeRegionId(res), static_cast<u8>(class_id)); return res; } @@ -278,21 +294,25 @@ class SizeClassAllocator32 { return &size_class_info_array[class_id]; } - void PopulateFreeList(AllocatorStats *stat, AllocatorCache *c, + bool PopulateFreeList(AllocatorStats *stat, AllocatorCache *c, SizeClassInfo *sci, uptr class_id) { uptr size = ClassIdToSize(class_id); uptr reg = AllocateRegion(stat, class_id); + if (UNLIKELY(!reg)) + return false; uptr n_chunks = kRegionSize / (size + kMetadataSize); uptr max_count = TransferBatch::MaxCached(class_id); + CHECK_GT(max_count, 0); TransferBatch *b = nullptr; for (uptr i = reg; i < reg + n_chunks * size; i += size) { if (!b) { b = c->CreateBatch(class_id, this, (TransferBatch*)i); + if (UNLIKELY(!b)) + return false; b->Clear(); } b->Add((void*)i); if (b->Count() == max_count) { - CHECK_GT(b->Count(), 0); sci->free_list.push_back(b); b = nullptr; } @@ -301,10 +321,9 @@ class SizeClassAllocator32 { CHECK_GT(b->Count(), 0); sci->free_list.push_back(b); } + return true; } ByteMap possible_regions; SizeClassInfo size_class_info_array[kNumClasses]; }; - - diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h index 035d92b..0c2e72c 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h @@ -80,7 +80,7 @@ class SizeClassAllocator64 { CHECK_NE(NonConstSpaceBeg, ~(uptr)0); } SetReleaseToOSIntervalMs(release_to_os_interval_ms); - MapWithCallback(SpaceEnd(), AdditionalSize()); + MapWithCallbackOrDie(SpaceEnd(), AdditionalSize()); } s32 ReleaseToOSIntervalMs() const { @@ -92,16 +92,6 @@ class SizeClassAllocator64 { memory_order_relaxed); } - void MapWithCallback(uptr beg, uptr size) { - CHECK_EQ(beg, reinterpret_cast<uptr>(MmapFixedOrDie(beg, size))); - MapUnmapCallback().OnMap(beg, size); - } - - void UnmapWithCallback(uptr beg, uptr size) { - MapUnmapCallback().OnUnmap(beg, size); - UnmapOrDie(reinterpret_cast<void *>(beg), size); - } - static bool CanAllocate(uptr size, uptr alignment) { return size <= SizeClassMap::kMaxSize && alignment <= SizeClassMap::kMaxSize; @@ -116,16 +106,20 @@ class SizeClassAllocator64 { BlockingMutexLock l(®ion->mutex); uptr old_num_chunks = region->num_freed_chunks; uptr new_num_freed_chunks = old_num_chunks + n_chunks; - EnsureFreeArraySpace(region, region_beg, new_num_freed_chunks); + // Failure to allocate free array space while releasing memory is non + // recoverable. + if (UNLIKELY(!EnsureFreeArraySpace(region, region_beg, + new_num_freed_chunks))) + DieOnFailure::OnOOM(); for (uptr i = 0; i < n_chunks; i++) free_array[old_num_chunks + i] = chunks[i]; region->num_freed_chunks = new_num_freed_chunks; - region->n_freed += n_chunks; + region->stats.n_freed += n_chunks; MaybeReleaseToOS(class_id); } - NOINLINE void GetFromAllocator(AllocatorStats *stat, uptr class_id, + NOINLINE bool GetFromAllocator(AllocatorStats *stat, uptr class_id, CompactPtrT *chunks, uptr n_chunks) { RegionInfo *region = GetRegionInfo(class_id); uptr region_beg = GetRegionBeginBySizeClass(class_id); @@ -133,18 +127,19 @@ class SizeClassAllocator64 { BlockingMutexLock l(®ion->mutex); if (UNLIKELY(region->num_freed_chunks < n_chunks)) { - PopulateFreeArray(stat, class_id, region, - n_chunks - region->num_freed_chunks); + if (UNLIKELY(!PopulateFreeArray(stat, class_id, region, + n_chunks - region->num_freed_chunks))) + return false; CHECK_GE(region->num_freed_chunks, n_chunks); } region->num_freed_chunks -= n_chunks; uptr base_idx = region->num_freed_chunks; for (uptr i = 0; i < n_chunks; i++) chunks[i] = free_array[base_idx + i]; - region->n_allocated += n_chunks; + region->stats.n_allocated += n_chunks; + return true; } - bool PointerIsMine(const void *p) { uptr P = reinterpret_cast<uptr>(p); if (kUsingConstantSpaceBeg && (kSpaceBeg % kSpaceSize) == 0) @@ -211,7 +206,7 @@ class SizeClassAllocator64 { // Test-only. void TestOnlyUnmap() { - UnmapWithCallback(SpaceBeg(), kSpaceSize + AdditionalSize()); + UnmapWithCallbackOrDie(SpaceBeg(), kSpaceSize + AdditionalSize()); } static void FillMemoryProfile(uptr start, uptr rss, bool file, uptr *stats, @@ -224,15 +219,15 @@ class SizeClassAllocator64 { void PrintStats(uptr class_id, uptr rss) { RegionInfo *region = GetRegionInfo(class_id); if (region->mapped_user == 0) return; - uptr in_use = region->n_allocated - region->n_freed; + uptr in_use = region->stats.n_allocated - region->stats.n_freed; uptr avail_chunks = region->allocated_user / ClassIdToSize(class_id); Printf( - " %02zd (%6zd): mapped: %6zdK allocs: %7zd frees: %7zd inuse: %6zd " + "%s %02zd (%6zd): mapped: %6zdK allocs: %7zd frees: %7zd inuse: %6zd " "num_freed_chunks %7zd avail: %6zd rss: %6zdK releases: %6zd\n", - class_id, ClassIdToSize(class_id), region->mapped_user >> 10, - region->n_allocated, region->n_freed, in_use, - region->num_freed_chunks, avail_chunks, rss >> 10, - region->rtoi.num_releases); + region->exhausted ? "F" : " ", class_id, ClassIdToSize(class_id), + region->mapped_user >> 10, region->stats.n_allocated, + region->stats.n_freed, in_use, region->num_freed_chunks, avail_chunks, + rss >> 10, region->rtoi.num_releases); } void PrintStats() { @@ -242,8 +237,8 @@ class SizeClassAllocator64 { for (uptr class_id = 1; class_id < kNumClasses; class_id++) { RegionInfo *region = GetRegionInfo(class_id); total_mapped += region->mapped_user; - n_allocated += region->n_allocated; - n_freed += region->n_freed; + n_allocated += region->stats.n_allocated; + n_freed += region->stats.n_freed; } Printf("Stats: SizeClassAllocator64: %zdM mapped in %zd allocations; " "remains %zd\n", @@ -326,6 +321,11 @@ class SizeClassAllocator64 { atomic_sint32_t release_to_os_interval_ms_; + struct Stats { + uptr n_allocated; + uptr n_freed; + }; + struct ReleaseToOsInfo { uptr n_freed_at_last_release; uptr num_releases; @@ -340,8 +340,9 @@ class SizeClassAllocator64 { uptr allocated_meta; // Bytes allocated for metadata. uptr mapped_user; // Bytes mapped for user memory. uptr mapped_meta; // Bytes mapped for metadata. - u32 rand_state; // Seed for random shuffle, used if kRandomShuffleChunks. - uptr n_allocated, n_freed; // Just stats. + u32 rand_state; // Seed for random shuffle, used if kRandomShuffleChunks. + bool exhausted; // Whether region is out of space for new chunks. + Stats stats; ReleaseToOsInfo rtoi; }; COMPILER_CHECK(sizeof(RegionInfo) >= kCacheLineSize); @@ -386,7 +387,26 @@ class SizeClassAllocator64 { kFreeArraySize); } - void EnsureFreeArraySpace(RegionInfo *region, uptr region_beg, + bool MapWithCallback(uptr beg, uptr size) { + uptr mapped = reinterpret_cast<uptr>(MmapFixedOrDieOnFatalError(beg, size)); + if (UNLIKELY(!mapped)) + return false; + CHECK_EQ(beg, mapped); + MapUnmapCallback().OnMap(beg, size); + return true; + } + + void MapWithCallbackOrDie(uptr beg, uptr size) { + CHECK_EQ(beg, reinterpret_cast<uptr>(MmapFixedOrDie(beg, size))); + MapUnmapCallback().OnMap(beg, size); + } + + void UnmapWithCallbackOrDie(uptr beg, uptr size) { + MapUnmapCallback().OnUnmap(beg, size); + UnmapOrDie(reinterpret_cast<void *>(beg), size); + } + + bool EnsureFreeArraySpace(RegionInfo *region, uptr region_beg, uptr num_freed_chunks) { uptr needed_space = num_freed_chunks * sizeof(CompactPtrT); if (region->mapped_free_array < needed_space) { @@ -395,66 +415,87 @@ class SizeClassAllocator64 { uptr current_map_end = reinterpret_cast<uptr>(GetFreeArray(region_beg)) + region->mapped_free_array; uptr new_map_size = new_mapped_free_array - region->mapped_free_array; - MapWithCallback(current_map_end, new_map_size); + if (UNLIKELY(!MapWithCallback(current_map_end, new_map_size))) + return false; region->mapped_free_array = new_mapped_free_array; } + return true; } - - NOINLINE void PopulateFreeArray(AllocatorStats *stat, uptr class_id, + NOINLINE bool PopulateFreeArray(AllocatorStats *stat, uptr class_id, RegionInfo *region, uptr requested_count) { // region->mutex is held. - uptr size = ClassIdToSize(class_id); - uptr beg_idx = region->allocated_user; - uptr end_idx = beg_idx + requested_count * size; - uptr region_beg = GetRegionBeginBySizeClass(class_id); - if (end_idx > region->mapped_user) { + const uptr size = ClassIdToSize(class_id); + const uptr new_space_beg = region->allocated_user; + const uptr new_space_end = new_space_beg + requested_count * size; + const uptr region_beg = GetRegionBeginBySizeClass(class_id); + + // Map more space for chunks, if necessary. + if (new_space_end > region->mapped_user) { if (!kUsingConstantSpaceBeg && region->mapped_user == 0) region->rand_state = static_cast<u32>(region_beg >> 12); // From ASLR. // Do the mmap for the user memory. uptr map_size = kUserMapSize; - while (end_idx > region->mapped_user + map_size) + while (new_space_end > region->mapped_user + map_size) map_size += kUserMapSize; - CHECK_GE(region->mapped_user + map_size, end_idx); - MapWithCallback(region_beg + region->mapped_user, map_size); + CHECK_GE(region->mapped_user + map_size, new_space_end); + if (UNLIKELY(!MapWithCallback(region_beg + region->mapped_user, + map_size))) + return false; stat->Add(AllocatorStatMapped, map_size); region->mapped_user += map_size; } - CompactPtrT *free_array = GetFreeArray(region_beg); - uptr total_count = (region->mapped_user - beg_idx) / size; - uptr num_freed_chunks = region->num_freed_chunks; - EnsureFreeArraySpace(region, region_beg, num_freed_chunks + total_count); - for (uptr i = 0; i < total_count; i++) { - uptr chunk = beg_idx + i * size; - free_array[num_freed_chunks + total_count - 1 - i] = - PointerToCompactPtr(0, chunk); + const uptr new_chunks_count = (region->mapped_user - new_space_beg) / size; + + // Calculate the required space for metadata. + const uptr requested_allocated_meta = + region->allocated_meta + new_chunks_count * kMetadataSize; + uptr requested_mapped_meta = region->mapped_meta; + while (requested_allocated_meta > requested_mapped_meta) + requested_mapped_meta += kMetaMapSize; + // Check whether this size class is exhausted. + if (region->mapped_user + requested_mapped_meta > + kRegionSize - kFreeArraySize) { + if (!region->exhausted) { + region->exhausted = true; + Printf("%s: Out of memory. ", SanitizerToolName); + Printf("The process has exhausted %zuMB for size class %zu.\n", + kRegionSize >> 20, size); + } + return false; + } + // Map more space for metadata, if necessary. + if (requested_mapped_meta > region->mapped_meta) { + if (UNLIKELY(!MapWithCallback( + GetMetadataEnd(region_beg) - requested_mapped_meta, + requested_mapped_meta - region->mapped_meta))) + return false; + region->mapped_meta = requested_mapped_meta; } + + // If necessary, allocate more space for the free array and populate it with + // newly allocated chunks. + const uptr total_freed_chunks = region->num_freed_chunks + new_chunks_count; + if (UNLIKELY(!EnsureFreeArraySpace(region, region_beg, total_freed_chunks))) + return false; + CompactPtrT *free_array = GetFreeArray(region_beg); + for (uptr i = 0, chunk = new_space_beg; i < new_chunks_count; + i++, chunk += size) + free_array[total_freed_chunks - 1 - i] = PointerToCompactPtr(0, chunk); if (kRandomShuffleChunks) - RandomShuffle(&free_array[num_freed_chunks], total_count, + RandomShuffle(&free_array[region->num_freed_chunks], new_chunks_count, ®ion->rand_state); - region->num_freed_chunks += total_count; - region->allocated_user += total_count * size; - CHECK_LE(region->allocated_user, region->mapped_user); - region->allocated_meta += total_count * kMetadataSize; - if (region->allocated_meta > region->mapped_meta) { - uptr map_size = kMetaMapSize; - while (region->allocated_meta > region->mapped_meta + map_size) - map_size += kMetaMapSize; - // Do the mmap for the metadata. - CHECK_GE(region->mapped_meta + map_size, region->allocated_meta); - MapWithCallback(GetMetadataEnd(region_beg) - - region->mapped_meta - map_size, map_size); - region->mapped_meta += map_size; - } + // All necessary memory is mapped and now it is safe to advance all + // 'allocated_*' counters. + region->num_freed_chunks += new_chunks_count; + region->allocated_user += new_chunks_count * size; + CHECK_LE(region->allocated_user, region->mapped_user); + region->allocated_meta = requested_allocated_meta; CHECK_LE(region->allocated_meta, region->mapped_meta); - if (region->mapped_user + region->mapped_meta > - kRegionSize - kFreeArraySize) { - Printf("%s: Out of memory. Dying. ", SanitizerToolName); - Printf("The process has exhausted %zuMB for size class %zu.\n", - kRegionSize / 1024 / 1024, size); - Die(); - } + region->exhausted = false; + + return true; } void MaybeReleaseChunkRange(uptr region_beg, uptr chunk_size, @@ -478,8 +519,8 @@ class SizeClassAllocator64 { uptr n = region->num_freed_chunks; if (n * chunk_size < page_size) return; // No chance to release anything. - if ((region->n_freed - region->rtoi.n_freed_at_last_release) * chunk_size < - page_size) { + if ((region->stats.n_freed - + region->rtoi.n_freed_at_last_release) * chunk_size < page_size) { return; // Nothing new to release. } @@ -508,7 +549,7 @@ class SizeClassAllocator64 { CHECK_GT(chunk - prev, scaled_chunk_size); if (prev + scaled_chunk_size - range_beg >= kScaledGranularity) { MaybeReleaseChunkRange(region_beg, chunk_size, range_beg, prev); - region->rtoi.n_freed_at_last_release = region->n_freed; + region->rtoi.n_freed_at_last_release = region->stats.n_freed; region->rtoi.num_releases++; } range_beg = chunk; @@ -517,5 +558,3 @@ class SizeClassAllocator64 { } } }; - - diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_secondary.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_secondary.h index 2e98e59..261dfb5 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_secondary.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_allocator_secondary.h @@ -17,17 +17,19 @@ // This class can (de)allocate only large chunks of memory using mmap/unmap. // The main purpose of this allocator is to cover large and rare allocation // sizes not covered by more efficient allocators (e.g. SizeClassAllocator64). -template <class MapUnmapCallback = NoOpMapUnmapCallback> +template <class MapUnmapCallback = NoOpMapUnmapCallback, + class FailureHandlerT = ReturnNullOrDieOnFailure> class LargeMmapAllocator { public: - void InitLinkerInitialized(bool may_return_null) { + typedef FailureHandlerT FailureHandler; + + void InitLinkerInitialized() { page_size_ = GetPageSizeCached(); - atomic_store(&may_return_null_, may_return_null, memory_order_relaxed); } - void Init(bool may_return_null) { + void Init() { internal_memset(this, 0, sizeof(*this)); - InitLinkerInitialized(may_return_null); + InitLinkerInitialized(); } void *Allocate(AllocatorStats *stat, uptr size, uptr alignment) { @@ -36,9 +38,12 @@ class LargeMmapAllocator { if (alignment > page_size_) map_size += alignment; // Overflow. - if (map_size < size) return ReturnNullOrDieOnBadRequest(); + if (map_size < size) + return FailureHandler::OnBadRequest(); uptr map_beg = reinterpret_cast<uptr>( - MmapOrDie(map_size, "LargeMmapAllocator")); + MmapOrDieOnFatalError(map_size, "LargeMmapAllocator")); + if (!map_beg) + return FailureHandler::OnOOM(); CHECK(IsAligned(map_beg, page_size_)); MapUnmapCallback().OnMap(map_beg, map_size); uptr map_end = map_beg + map_size; @@ -72,24 +77,6 @@ class LargeMmapAllocator { return reinterpret_cast<void*>(res); } - bool MayReturnNull() const { - return atomic_load(&may_return_null_, memory_order_acquire); - } - - void *ReturnNullOrDieOnBadRequest() { - if (MayReturnNull()) return nullptr; - ReportAllocatorCannotReturnNull(false); - } - - void *ReturnNullOrDieOnOOM() { - if (MayReturnNull()) return nullptr; - ReportAllocatorCannotReturnNull(true); - } - - void SetMayReturnNull(bool may_return_null) { - atomic_store(&may_return_null_, may_return_null, memory_order_release); - } - void Deallocate(AllocatorStats *stat, void *p) { Header *h = GetHeader(p); { @@ -275,7 +262,6 @@ class LargeMmapAllocator { struct Stats { uptr n_allocs, n_frees, currently_allocated, max_allocated, by_size_log[64]; } stats; - atomic_uint8_t may_return_null_; SpinMutex mutex_; }; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang.h index 38363e8..65b3a38 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang.h @@ -71,16 +71,25 @@ INLINE typename T::Type atomic_exchange(volatile T *a, return v; } -template<typename T> -INLINE bool atomic_compare_exchange_strong(volatile T *a, - typename T::Type *cmp, +template <typename T> +INLINE bool atomic_compare_exchange_strong(volatile T *a, typename T::Type *cmp, typename T::Type xchg, memory_order mo) { typedef typename T::Type Type; Type cmpv = *cmp; - Type prev = __sync_val_compare_and_swap(&a->val_dont_use, cmpv, xchg); - if (prev == cmpv) - return true; + Type prev; +#if defined(_MIPS_SIM) && _MIPS_SIM == _ABIO32 + if (sizeof(*a) == 8) { + Type volatile *val_ptr = const_cast<Type volatile *>(&a->val_dont_use); + prev = __mips_sync_val_compare_and_swap<u64>( + reinterpret_cast<u64 volatile *>(val_ptr), (u64)cmpv, (u64)xchg); + } else { + prev = __sync_val_compare_and_swap(&a->val_dont_use, cmpv, xchg); + } +#else + prev = __sync_val_compare_and_swap(&a->val_dont_use, cmpv, xchg); +#endif + if (prev == cmpv) return true; *cmp = prev; return false; } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang_other.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang_other.h index 099b9f7..d2acc31 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang_other.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_atomic_clang_other.h @@ -17,6 +17,56 @@ namespace __sanitizer { +// MIPS32 does not support atomic > 4 bytes. To address this lack of +// functionality, the sanitizer library provides helper methods which use an +// internal spin lock mechanism to emulate atomic oprations when the size is +// 8 bytes. +#if defined(_MIPS_SIM) && _MIPS_SIM == _ABIO32 +static void __spin_lock(volatile int *lock) { + while (__sync_lock_test_and_set(lock, 1)) + while (*lock) { + } +} + +static void __spin_unlock(volatile int *lock) { __sync_lock_release(lock); } + + +// Make sure the lock is on its own cache line to prevent false sharing. +// Put it inside a struct that is aligned and padded to the typical MIPS +// cacheline which is 32 bytes. +static struct { + int lock; + char pad[32 - sizeof(int)]; +} __attribute__((aligned(32))) lock = {0}; + +template <class T> +T __mips_sync_fetch_and_add(volatile T *ptr, T val) { + T ret; + + __spin_lock(&lock.lock); + + ret = *ptr; + *ptr = ret + val; + + __spin_unlock(&lock.lock); + + return ret; +} + +template <class T> +T __mips_sync_val_compare_and_swap(volatile T *ptr, T oldval, T newval) { + T ret; + __spin_lock(&lock.lock); + + ret = *ptr; + if (ret == oldval) *ptr = newval; + + __spin_unlock(&lock.lock); + + return ret; +} +#endif + INLINE void proc_yield(int cnt) { __asm__ __volatile__("" ::: "memory"); } @@ -53,8 +103,15 @@ INLINE typename T::Type atomic_load( // 64-bit load on 32-bit platform. // Gross, but simple and reliable. // Assume that it is not in read-only memory. +#if defined(_MIPS_SIM) && _MIPS_SIM == _ABIO32 + typename T::Type volatile *val_ptr = + const_cast<typename T::Type volatile *>(&a->val_dont_use); + v = __mips_sync_fetch_and_add<u64>( + reinterpret_cast<u64 volatile *>(val_ptr), 0); +#else v = __sync_fetch_and_add( const_cast<typename T::Type volatile *>(&a->val_dont_use), 0); +#endif } return v; } @@ -84,7 +141,14 @@ INLINE void atomic_store(volatile T *a, typename T::Type v, memory_order mo) { typename T::Type cmp = a->val_dont_use; typename T::Type cur; for (;;) { +#if defined(_MIPS_SIM) && _MIPS_SIM == _ABIO32 + typename T::Type volatile *val_ptr = + const_cast<typename T::Type volatile *>(&a->val_dont_use); + cur = __mips_sync_val_compare_and_swap<u64>( + reinterpret_cast<u64 volatile *>(val_ptr), (u64)cmp, (u64)v); +#else cur = __sync_val_compare_and_swap(&a->val_dont_use, cmp, v); +#endif if (cmp == v) break; cmp = cur; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.cc index 9824a51..7e6f8fc 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.cc @@ -199,23 +199,24 @@ const char *StripModuleName(const char *module) { return module; } -void ReportErrorSummary(const char *error_message) { +void ReportErrorSummary(const char *error_message, const char *alt_tool_name) { if (!common_flags()->print_summary) return; InternalScopedString buff(kMaxSummaryLength); - buff.append("SUMMARY: %s: %s", SanitizerToolName, error_message); + buff.append("SUMMARY: %s: %s", + alt_tool_name ? alt_tool_name : SanitizerToolName, error_message); __sanitizer_report_error_summary(buff.data()); } #if !SANITIZER_GO -void ReportErrorSummary(const char *error_type, const AddressInfo &info) { - if (!common_flags()->print_summary) - return; +void ReportErrorSummary(const char *error_type, const AddressInfo &info, + const char *alt_tool_name) { + if (!common_flags()->print_summary) return; InternalScopedString buff(kMaxSummaryLength); buff.append("%s ", error_type); RenderFrame(&buff, "%L %F", 0, info, common_flags()->symbolize_vs_style, common_flags()->strip_path_prefix); - ReportErrorSummary(buff.data()); + ReportErrorSummary(buff.data(), alt_tool_name); } #endif @@ -283,9 +284,10 @@ void LoadedModule::clear() { } } -void LoadedModule::addAddressRange(uptr beg, uptr end, bool executable) { +void LoadedModule::addAddressRange(uptr beg, uptr end, bool executable, + bool writable) { void *mem = InternalAlloc(sizeof(AddressRange)); - AddressRange *r = new(mem) AddressRange(beg, end, executable); + AddressRange *r = new(mem) AddressRange(beg, end, executable, writable); ranges_.push_back(r); if (executable && end > max_executable_address_) max_executable_address_ = end; @@ -489,7 +491,8 @@ void __sanitizer_set_report_fd(void *fd) { report_file.fd_pid = internal_getpid(); } -void __sanitizer_report_error_summary(const char *error_summary) { +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_report_error_summary, + const char *error_summary) { Printf("%s\n", error_summary); } @@ -504,11 +507,4 @@ int __sanitizer_install_malloc_and_free_hooks(void (*malloc_hook)(const void *, void (*free_hook)(const void *)) { return InstallMallocFreeHooks(malloc_hook, free_hook); } - -#if !SANITIZER_GO && !SANITIZER_SUPPORTS_WEAK_HOOKS -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_print_memory_profile(int top_percent) { - (void)top_percent; -} -#endif } // extern "C" diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.h index 2dabb50..89aae57 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common.h @@ -72,7 +72,7 @@ INLINE uptr GetPageSizeCached() { uptr GetMmapGranularity(); uptr GetMaxVirtualAddress(); // Threads -uptr GetTid(); +tid_t GetTid(); uptr GetThreadSelf(); void GetThreadStackTopAndBottom(bool at_initialization, uptr *stack_top, uptr *stack_bottom); @@ -85,21 +85,30 @@ INLINE void *MmapOrDieQuietly(uptr size, const char *mem_type) { return MmapOrDie(size, mem_type, /*raw_report*/ true); } void UnmapOrDie(void *addr, uptr size); +// Behaves just like MmapOrDie, but tolerates out of memory condition, in that +// case returns nullptr. +void *MmapOrDieOnFatalError(uptr size, const char *mem_type); void *MmapFixedNoReserve(uptr fixed_addr, uptr size, const char *name = nullptr); void *MmapNoReserveOrDie(uptr size, const char *mem_type); void *MmapFixedOrDie(uptr fixed_addr, uptr size); +// Behaves just like MmapFixedOrDie, but tolerates out of memory condition, in +// that case returns nullptr. +void *MmapFixedOrDieOnFatalError(uptr fixed_addr, uptr size); void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name = nullptr); void *MmapNoAccess(uptr size); // Map aligned chunk of address space; size and alignment are powers of two. -void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type); +// Dies on all but out of memory errors, in the latter case returns nullptr. +void *MmapAlignedOrDieOnFatalError(uptr size, uptr alignment, + const char *mem_type); // Disallow access to a memory range. Use MmapFixedNoAccess to allocate an // unaccessible memory. bool MprotectNoAccess(uptr addr, uptr size); bool MprotectReadOnly(uptr addr, uptr size); // Find an available address space. -uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding); +uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding, + uptr *largest_gap_found); // Used to check if we can map shadow memory to a fixed location. bool MemoryRangeIsAvailable(uptr range_start, uptr range_end); @@ -317,15 +326,9 @@ bool AddressSpaceIsUnlimited(); void SetAddressSpaceUnlimited(); void AdjustStackSize(void *attr); void PrepareForSandboxing(__sanitizer_sandbox_arguments *args); -void CovPrepareForSandboxing(__sanitizer_sandbox_arguments *args); void SetSandboxingCallback(void (*f)()); -void CoverageUpdateMapping(); -void CovBeforeFork(); -void CovAfterFork(int child_pid); - void InitializeCoverage(bool enabled, const char *coverage_dir); -void ReInitializeCoverage(bool enabled, const char *coverage_dir); void InitTlsSize(); uptr GetTlsSize(); @@ -380,8 +383,9 @@ void SetSoftRssLimitExceededCallback(void (*Callback)(bool exceeded)); // Functions related to signal handling. typedef void (*SignalHandlerType)(int, void *, void *); -bool IsHandledDeadlySignal(int signum); +HandleSignalMode GetHandleSignalMode(int signum); void InstallDeadlySignalHandlers(SignalHandlerType handler); +const char *DescribeSignalOrException(int signo); // Alternative signal stack (POSIX-only). void SetAlternateSignalStack(); void UnsetAlternateSignalStack(); @@ -391,12 +395,16 @@ const int kMaxSummaryLength = 1024; // Construct a one-line string: // SUMMARY: SanitizerToolName: error_message // and pass it to __sanitizer_report_error_summary. -void ReportErrorSummary(const char *error_message); +// If alt_tool_name is provided, it's used in place of SanitizerToolName. +void ReportErrorSummary(const char *error_message, + const char *alt_tool_name = nullptr); // Same as above, but construct error_message as: // error_type file:line[:column][ function] -void ReportErrorSummary(const char *error_type, const AddressInfo &info); +void ReportErrorSummary(const char *error_type, const AddressInfo &info, + const char *alt_tool_name = nullptr); // Same as above, but obtains AddressInfo by symbolizing top stack trace frame. -void ReportErrorSummary(const char *error_type, const StackTrace *trace); +void ReportErrorSummary(const char *error_type, const StackTrace *trace, + const char *alt_tool_name = nullptr); // Math #if SANITIZER_WINDOWS && !defined(__clang__) && !defined(__GNUC__) @@ -712,7 +720,7 @@ class LoadedModule { void set(const char *module_name, uptr base_address, ModuleArch arch, u8 uuid[kModuleUUIDSize], bool instrumented); void clear(); - void addAddressRange(uptr beg, uptr end, bool executable); + void addAddressRange(uptr beg, uptr end, bool executable, bool writable); bool containsAddress(uptr address) const; const char *full_name() const { return full_name_; } @@ -727,9 +735,14 @@ class LoadedModule { uptr beg; uptr end; bool executable; - - AddressRange(uptr beg, uptr end, bool executable) - : next(nullptr), beg(beg), end(end), executable(executable) {} + bool writable; + + AddressRange(uptr beg, uptr end, bool executable, bool writable) + : next(nullptr), + beg(beg), + end(end), + executable(executable), + writable(writable) {} }; const IntrusiveList<AddressRange> &ranges() const { return ranges_; } @@ -801,8 +814,11 @@ INLINE void LogMessageOnPrintf(const char *str) {} #if SANITIZER_LINUX // Initialize Android logging. Any writes before this are silently lost. void AndroidLogInit(); +void SetAbortMessage(const char *); #else INLINE void AndroidLogInit() {} +// FIXME: MacOS implementation could use CRSetCrashLogMessage. +INLINE void SetAbortMessage(const char *) {} #endif #if SANITIZER_ANDROID @@ -910,6 +926,12 @@ struct StackDepotStats { // indicate that sanitizer allocator should not attempt to release memory to OS. const s32 kReleaseToOSIntervalNever = -1; +void CheckNoDeepBind(const char *filename, int flag); + +// Returns the requested amount of random data (up to 256 bytes) that can then +// be used to seed a PRNG. +bool GetRandom(void *buffer, uptr length); + } // namespace __sanitizer inline void *operator new(__sanitizer::operator_new_size_type size, diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc index ca571d1..8607bf4 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc @@ -24,7 +24,8 @@ // COMMON_INTERCEPTOR_SET_THREAD_NAME // COMMON_INTERCEPTOR_ON_DLOPEN // COMMON_INTERCEPTOR_ON_EXIT -// COMMON_INTERCEPTOR_MUTEX_LOCK +// COMMON_INTERCEPTOR_MUTEX_PRE_LOCK +// COMMON_INTERCEPTOR_MUTEX_POST_LOCK // COMMON_INTERCEPTOR_MUTEX_UNLOCK // COMMON_INTERCEPTOR_MUTEX_REPAIR // COMMON_INTERCEPTOR_SET_PTHREAD_NAME @@ -33,10 +34,13 @@ // COMMON_INTERCEPTOR_MEMSET_IMPL // COMMON_INTERCEPTOR_MEMMOVE_IMPL // COMMON_INTERCEPTOR_MEMCPY_IMPL +// COMMON_INTERCEPTOR_COPY_STRING +// COMMON_INTERCEPTOR_STRNDUP_IMPL //===----------------------------------------------------------------------===// #include "interception/interception.h" #include "sanitizer_addrhashmap.h" +#include "sanitizer_errno.h" #include "sanitizer_placement_new.h" #include "sanitizer_platform_interceptors.h" #include "sanitizer_tls_get_addr.h" @@ -44,15 +48,9 @@ #include <stdarg.h> #if SANITIZER_INTERCEPTOR_HOOKS -#define CALL_WEAK_INTERCEPTOR_HOOK(f, ...) \ - do { \ - if (f) \ - f(__VA_ARGS__); \ - } while (false); -#define DECLARE_WEAK_INTERCEPTOR_HOOK(f, ...) \ - extern "C" { \ - SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void f(__VA_ARGS__); \ - } // extern "C" +#define CALL_WEAK_INTERCEPTOR_HOOK(f, ...) f(__VA_ARGS__); +#define DECLARE_WEAK_INTERCEPTOR_HOOK(f, ...) \ + SANITIZER_INTERFACE_WEAK_DEF(void, f, __VA_ARGS__) {} #else #define DECLARE_WEAK_INTERCEPTOR_HOOK(f, ...) #define CALL_WEAK_INTERCEPTOR_HOOK(f, ...) @@ -95,8 +93,12 @@ bool PlatformHasDifferentMemcpyAndMemmove(); #define COMMON_INTERCEPTOR_FD_ACCESS(ctx, fd) {} #endif -#ifndef COMMON_INTERCEPTOR_MUTEX_LOCK -#define COMMON_INTERCEPTOR_MUTEX_LOCK(ctx, m) {} +#ifndef COMMON_INTERCEPTOR_MUTEX_PRE_LOCK +#define COMMON_INTERCEPTOR_MUTEX_PRE_LOCK(ctx, m) {} +#endif + +#ifndef COMMON_INTERCEPTOR_MUTEX_POST_LOCK +#define COMMON_INTERCEPTOR_MUTEX_POST_LOCK(ctx, m) {} #endif #ifndef COMMON_INTERCEPTOR_MUTEX_UNLOCK @@ -140,15 +142,13 @@ bool PlatformHasDifferentMemcpyAndMemmove(); #define COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED (0) #endif -#define COMMON_INTERCEPTOR_READ_STRING_OF_LEN(ctx, s, len, n) \ - COMMON_INTERCEPTOR_READ_RANGE((ctx), (s), \ - common_flags()->strict_string_checks ? (len) + 1 : (n) ) - #define COMMON_INTERCEPTOR_READ_STRING(ctx, s, n) \ - COMMON_INTERCEPTOR_READ_STRING_OF_LEN((ctx), (s), REAL(strlen)(s), (n)) + COMMON_INTERCEPTOR_READ_RANGE((ctx), (s), \ + common_flags()->strict_string_checks ? (REAL(strlen)(s)) + 1 : (n) ) #ifndef COMMON_INTERCEPTOR_ON_DLOPEN -#define COMMON_INTERCEPTOR_ON_DLOPEN(filename, flag) {} +#define COMMON_INTERCEPTOR_ON_DLOPEN(filename, flag) \ + CheckNoDeepBind(filename, flag); #endif #ifndef COMMON_INTERCEPTOR_GET_TLS_RANGE @@ -220,6 +220,24 @@ bool PlatformHasDifferentMemcpyAndMemmove(); } #endif +#ifndef COMMON_INTERCEPTOR_COPY_STRING +#define COMMON_INTERCEPTOR_COPY_STRING(ctx, to, from, size) {} +#endif + +#ifndef COMMON_INTERCEPTOR_STRNDUP_IMPL +#define COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size) \ + COMMON_INTERCEPTOR_ENTER(ctx, strndup, s, size); \ + uptr copy_length = internal_strnlen(s, size); \ + char *new_mem = (char *)WRAP(malloc)(copy_length + 1); \ + if (common_flags()->intercept_strndup) { \ + COMMON_INTERCEPTOR_READ_STRING(ctx, s, Min(size, copy_length + 1)); \ + } \ + COMMON_INTERCEPTOR_COPY_STRING(ctx, new_mem, s, copy_length); \ + internal_memcpy(new_mem, s, copy_length); \ + new_mem[copy_length] = '\0'; \ + return new_mem; +#endif + struct FileMetadata { // For open_memstream(). char **addr; @@ -303,11 +321,31 @@ INTERCEPTOR(SIZE_T, strnlen, const char *s, SIZE_T maxlen) { #define INIT_STRNLEN #endif +#if SANITIZER_INTERCEPT_STRNDUP +INTERCEPTOR(char*, strndup, const char *s, uptr size) { + void *ctx; + COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size); +} +#define INIT_STRNDUP COMMON_INTERCEPT_FUNCTION(strndup) +#else +#define INIT_STRNDUP +#endif // SANITIZER_INTERCEPT_STRNDUP + +#if SANITIZER_INTERCEPT___STRNDUP +INTERCEPTOR(char*, __strndup, const char *s, uptr size) { + void *ctx; + COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size); +} +#define INIT___STRNDUP COMMON_INTERCEPT_FUNCTION(__strndup) +#else +#define INIT___STRNDUP +#endif // SANITIZER_INTERCEPT___STRNDUP + #if SANITIZER_INTERCEPT_TEXTDOMAIN INTERCEPTOR(char*, textdomain, const char *domainname) { void *ctx; COMMON_INTERCEPTOR_ENTER(ctx, textdomain, domainname); - COMMON_INTERCEPTOR_READ_STRING(ctx, domainname, 0); + if (domainname) COMMON_INTERCEPTOR_READ_STRING(ctx, domainname, 0); char *domain = REAL(textdomain)(domainname); if (domain) { COMMON_INTERCEPTOR_INITIALIZE_RANGE(domain, REAL(strlen)(domain) + 1); @@ -450,8 +488,7 @@ static inline void StrstrCheck(void *ctx, char *r, const char *s1, const char *s2) { uptr len1 = REAL(strlen)(s1); uptr len2 = REAL(strlen)(s2); - COMMON_INTERCEPTOR_READ_STRING_OF_LEN(ctx, s1, len1, - r ? r - s1 + len2 : len1 + 1); + COMMON_INTERCEPTOR_READ_STRING(ctx, s1, r ? r - s1 + len2 : len1 + 1); COMMON_INTERCEPTOR_READ_RANGE(ctx, s2, len2 + 1); } #endif @@ -500,6 +537,52 @@ INTERCEPTOR(char*, strcasestr, const char *s1, const char *s2) { #define INIT_STRCASESTR #endif +#if SANITIZER_INTERCEPT_STRTOK + +INTERCEPTOR(char*, strtok, char *str, const char *delimiters) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, strtok, str, delimiters); + if (!common_flags()->intercept_strtok) { + return REAL(strtok)(str, delimiters); + } + if (common_flags()->strict_string_checks) { + // If strict_string_checks is enabled, we check the whole first argument + // string on the first call (strtok saves this string in a static buffer + // for subsequent calls). We do not need to check strtok's result. + // As the delimiters can change, we check them every call. + if (str != nullptr) { + COMMON_INTERCEPTOR_READ_RANGE(ctx, str, REAL(strlen)(str) + 1); + } + COMMON_INTERCEPTOR_READ_RANGE(ctx, delimiters, + REAL(strlen)(delimiters) + 1); + return REAL(strtok)(str, delimiters); + } else { + // However, when strict_string_checks is disabled we cannot check the + // whole string on the first call. Instead, we check the result string + // which is guaranteed to be a NULL-terminated substring of the first + // argument. We also conservatively check one character of str and the + // delimiters. + if (str != nullptr) { + COMMON_INTERCEPTOR_READ_STRING(ctx, str, 1); + } + COMMON_INTERCEPTOR_READ_RANGE(ctx, delimiters, 1); + char *result = REAL(strtok)(str, delimiters); + if (result != nullptr) { + COMMON_INTERCEPTOR_READ_RANGE(ctx, result, REAL(strlen)(result) + 1); + } else if (str != nullptr) { + // No delimiter were found, it's safe to assume that the entire str was + // scanned. + COMMON_INTERCEPTOR_READ_RANGE(ctx, str, REAL(strlen)(str) + 1); + } + return result; + } +} + +#define INIT_STRTOK COMMON_INTERCEPT_FUNCTION(strtok) +#else +#define INIT_STRTOK +#endif + #if SANITIZER_INTERCEPT_MEMMEM DECLARE_WEAK_INTERCEPTOR_HOOK(__sanitizer_weak_hook_memmem, uptr called_pc, const void *s1, SIZE_T len1, const void *s2, @@ -531,10 +614,11 @@ INTERCEPTOR(char*, strchr, const char *s, int c) { return internal_strchr(s, c); COMMON_INTERCEPTOR_ENTER(ctx, strchr, s, c); char *result = REAL(strchr)(s, c); - uptr len = internal_strlen(s); - uptr n = result ? result - s + 1 : len + 1; - if (common_flags()->intercept_strchr) - COMMON_INTERCEPTOR_READ_STRING_OF_LEN(ctx, s, len, n); + if (common_flags()->intercept_strchr) { + // Keep strlen as macro argument, as macro may ignore it. + COMMON_INTERCEPTOR_READ_STRING(ctx, s, + (result ? result - s : REAL(strlen)(s)) + 1); + } return result; } #define INIT_STRCHR COMMON_INTERCEPT_FUNCTION(strchr) @@ -563,9 +647,8 @@ INTERCEPTOR(char*, strrchr, const char *s, int c) { if (COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED) return internal_strrchr(s, c); COMMON_INTERCEPTOR_ENTER(ctx, strrchr, s, c); - uptr len = internal_strlen(s); if (common_flags()->intercept_strchr) - COMMON_INTERCEPTOR_READ_STRING_OF_LEN(ctx, s, len, len + 1); + COMMON_INTERCEPTOR_READ_RANGE(ctx, s, REAL(strlen)(s) + 1); return REAL(strrchr)(s, c); } #define INIT_STRRCHR COMMON_INTERCEPT_FUNCTION(strrchr) @@ -842,6 +925,23 @@ INTERCEPTOR(SSIZE_T, read, int fd, void *ptr, SIZE_T count) { #define INIT_READ #endif +#if SANITIZER_INTERCEPT_FREAD +INTERCEPTOR(SIZE_T, fread, void *ptr, SIZE_T size, SIZE_T nmemb, void *file) { + // libc file streams can call user-supplied functions, see fopencookie. + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, fread, ptr, size, nmemb, file); + // FIXME: under ASan the call below may write to freed memory and corrupt + // its metadata. See + // https://github.com/google/sanitizers/issues/321. + SIZE_T res = REAL(fread)(ptr, size, nmemb, file); + if (res > 0) COMMON_INTERCEPTOR_WRITE_RANGE(ctx, ptr, res * size); + return res; +} +#define INIT_FREAD COMMON_INTERCEPT_FUNCTION(fread) +#else +#define INIT_FREAD +#endif + #if SANITIZER_INTERCEPT_PREAD INTERCEPTOR(SSIZE_T, pread, int fd, void *ptr, SIZE_T count, OFF_T offset) { void *ctx; @@ -942,6 +1042,20 @@ INTERCEPTOR(SSIZE_T, write, int fd, void *ptr, SIZE_T count) { #define INIT_WRITE #endif +#if SANITIZER_INTERCEPT_FWRITE +INTERCEPTOR(SIZE_T, fwrite, const void *p, uptr size, uptr nmemb, void *file) { + // libc file streams can call user-supplied functions, see fopencookie. + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, fwrite, p, size, nmemb, file); + SIZE_T res = REAL(fwrite)(p, size, nmemb, file); + if (res > 0) COMMON_INTERCEPTOR_READ_RANGE(ctx, p, res * size); + return res; +} +#define INIT_FWRITE COMMON_INTERCEPT_FUNCTION(fwrite) +#else +#define INIT_FWRITE +#endif + #if SANITIZER_INTERCEPT_PWRITE INTERCEPTOR(SSIZE_T, pwrite, int fd, void *ptr, SIZE_T count, OFF_T offset) { void *ctx; @@ -3251,6 +3365,30 @@ INTERCEPTOR(char *, strerror, int errnum) { #endif #if SANITIZER_INTERCEPT_STRERROR_R +// There are 2 versions of strerror_r: +// * POSIX version returns 0 on success, negative error code on failure, +// writes message to buf. +// * GNU version returns message pointer, which points to either buf or some +// static storage. +#if ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && !_GNU_SOURCE) || \ + SANITIZER_MAC || SANITIZER_ANDROID +// POSIX version. Spec is not clear on whether buf is NULL-terminated. +// At least on OSX, buf contents are valid even when the call fails. +INTERCEPTOR(int, strerror_r, int errnum, char *buf, SIZE_T buflen) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, strerror_r, errnum, buf, buflen); + // FIXME: under ASan the call below may write to freed memory and corrupt + // its metadata. See + // https://github.com/google/sanitizers/issues/321. + int res = REAL(strerror_r)(errnum, buf, buflen); + + SIZE_T sz = internal_strnlen(buf, buflen); + if (sz < buflen) ++sz; + COMMON_INTERCEPTOR_WRITE_RANGE(ctx, buf, sz); + return res; +} +#else +// GNU version. INTERCEPTOR(char *, strerror_r, int errnum, char *buf, SIZE_T buflen) { void *ctx; COMMON_INTERCEPTOR_ENTER(ctx, strerror_r, errnum, buf, buflen); @@ -3258,24 +3396,14 @@ INTERCEPTOR(char *, strerror_r, int errnum, char *buf, SIZE_T buflen) { // its metadata. See // https://github.com/google/sanitizers/issues/321. char *res = REAL(strerror_r)(errnum, buf, buflen); - // There are 2 versions of strerror_r: - // * POSIX version returns 0 on success, negative error code on failure, - // writes message to buf. - // * GNU version returns message pointer, which points to either buf or some - // static storage. - SIZE_T posix_res = (SIZE_T)res; - if (posix_res < 1024 || posix_res > (SIZE_T) - 1024) { - // POSIX version. Spec is not clear on whether buf is NULL-terminated. - // At least on OSX, buf contents are valid even when the call fails. - SIZE_T sz = internal_strnlen(buf, buflen); - if (sz < buflen) ++sz; - COMMON_INTERCEPTOR_WRITE_RANGE(ctx, buf, sz); - } else { - // GNU version. + if (res == buf) COMMON_INTERCEPTOR_WRITE_RANGE(ctx, res, REAL(strlen)(res) + 1); - } + else + COMMON_INTERCEPTOR_INITIALIZE_RANGE(res, REAL(strlen)(res) + 1); return res; } +#endif //(_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && !_GNU_SOURCE || + //SANITIZER_MAC #define INIT_STRERROR_R COMMON_INTERCEPT_FUNCTION(strerror_r); #else #define INIT_STRERROR_R @@ -3414,7 +3542,8 @@ INTERCEPTOR(int, getgroups, int size, u32 *lst) { // its metadata. See // https://github.com/google/sanitizers/issues/321. int res = REAL(getgroups)(size, lst); - if (res && lst) COMMON_INTERCEPTOR_WRITE_RANGE(ctx, lst, res * sizeof(*lst)); + if (res >= 0 && lst && size > 0) + COMMON_INTERCEPTOR_WRITE_RANGE(ctx, lst, res * sizeof(*lst)); return res; } #define INIT_GETGROUPS COMMON_INTERCEPT_FUNCTION(getgroups); @@ -3669,11 +3798,12 @@ INTERCEPTOR(void, _exit, int status) { INTERCEPTOR(int, pthread_mutex_lock, void *m) { void *ctx; COMMON_INTERCEPTOR_ENTER(ctx, pthread_mutex_lock, m); + COMMON_INTERCEPTOR_MUTEX_PRE_LOCK(ctx, m); int res = REAL(pthread_mutex_lock)(m); if (res == errno_EOWNERDEAD) COMMON_INTERCEPTOR_MUTEX_REPAIR(ctx, m); if (res == 0 || res == errno_EOWNERDEAD) - COMMON_INTERCEPTOR_MUTEX_LOCK(ctx, m); + COMMON_INTERCEPTOR_MUTEX_POST_LOCK(ctx, m); if (res == errno_EINVAL) COMMON_INTERCEPTOR_MUTEX_INVALID(ctx, m); return res; @@ -4547,7 +4677,7 @@ INTERCEPTOR(SIZE_T, iconv, void *cd, char **inbuf, SIZE_T *inbytesleft, // its metadata. See // https://github.com/google/sanitizers/issues/321. SIZE_T res = REAL(iconv)(cd, inbuf, inbytesleft, outbuf, outbytesleft); - if (res != (SIZE_T) - 1 && outbuf && *outbuf > outbuf_orig) { + if (outbuf && *outbuf > outbuf_orig) { SIZE_T sz = (char *)*outbuf - (char *)outbuf_orig; COMMON_INTERCEPTOR_WRITE_RANGE(ctx, outbuf_orig, sz); } @@ -4614,11 +4744,15 @@ void *__tls_get_addr_opt(void *arg); // descriptor offset as an argument instead of a pointer. GOT address // is passed in r12, so it's necessary to write it in assembly. This is // the function used by the compiler. -#define INIT_TLS_GET_ADDR COMMON_INTERCEPT_FUNCTION(__tls_get_addr_internal) +extern "C" uptr __tls_get_offset_wrapper(void *arg, uptr (*fn)(void *arg)); +#define INIT_TLS_GET_ADDR COMMON_INTERCEPT_FUNCTION(__tls_get_offset) +DEFINE_REAL(uptr, __tls_get_offset, void *arg) +extern "C" uptr __tls_get_offset(void *arg); +extern "C" uptr __interceptor___tls_get_offset(void *arg); INTERCEPTOR(uptr, __tls_get_addr_internal, void *arg) { void *ctx; COMMON_INTERCEPTOR_ENTER(ctx, __tls_get_addr_internal, arg); - uptr res = REAL(__tls_get_addr_internal)(arg); + uptr res = __tls_get_offset_wrapper(arg, REAL(__tls_get_offset)); uptr tp = reinterpret_cast<uptr>(__builtin_thread_pointer()); void *ptr = reinterpret_cast<void *>(res + tp); uptr tls_begin, tls_end; @@ -4630,32 +4764,43 @@ INTERCEPTOR(uptr, __tls_get_addr_internal, void *arg) { } return res; } -// We need a protected symbol aliasing the above, so that we can jump +// We need a hidden symbol aliasing the above, so that we can jump // directly to it from the assembly below. extern "C" __attribute__((alias("__interceptor___tls_get_addr_internal"), - visibility("protected"))) -uptr __interceptor___tls_get_addr_internal_protected(void *arg); + visibility("hidden"))) +uptr __tls_get_addr_hidden(void *arg); // Now carefully intercept __tls_get_offset. asm( ".text\n" - ".global __tls_get_offset\n" - "__tls_get_offset:\n" // The __intercept_ version has to exist, so that gen_dynamic_list.py // exports our symbol. + ".weak __tls_get_offset\n" + ".type __tls_get_offset, @function\n" + "__tls_get_offset:\n" ".global __interceptor___tls_get_offset\n" + ".type __interceptor___tls_get_offset, @function\n" "__interceptor___tls_get_offset:\n" #ifdef __s390x__ "la %r2, 0(%r2,%r12)\n" - "jg __interceptor___tls_get_addr_internal_protected\n" + "jg __tls_get_addr_hidden\n" #else "basr %r3,0\n" "0: la %r2,0(%r2,%r12)\n" "l %r4,1f-0b(%r3)\n" "b 0(%r4,%r3)\n" - "1: .long __interceptor___tls_get_addr_internal_protected - 0b\n" + "1: .long __tls_get_addr_hidden - 0b\n" #endif - ".type __tls_get_offset, @function\n" - ".size __tls_get_offset, .-__tls_get_offset\n" + ".size __interceptor___tls_get_offset, .-__interceptor___tls_get_offset\n" +// Assembly wrapper to call REAL(__tls_get_offset)(arg) + ".type __tls_get_offset_wrapper, @function\n" + "__tls_get_offset_wrapper:\n" +#ifdef __s390x__ + "sgr %r2,%r12\n" +#else + "sr %r2,%r12\n" +#endif + "br %r3\n" + ".size __tls_get_offset_wrapper, .-__tls_get_offset_wrapper\n" ); #endif // SANITIZER_S390 #else @@ -6026,6 +6171,86 @@ INTERCEPTOR(void *, getutxline, void *ut) { #define INIT_UTMPX #endif +#if SANITIZER_INTERCEPT_GETLOADAVG +INTERCEPTOR(int, getloadavg, double *loadavg, int nelem) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, getloadavg, loadavg, nelem); + int res = REAL(getloadavg)(loadavg, nelem); + if (res > 0) + COMMON_INTERCEPTOR_WRITE_RANGE(ctx, loadavg, res * sizeof(*loadavg)); + return res; +} +#define INIT_GETLOADAVG \ + COMMON_INTERCEPT_FUNCTION(getloadavg); +#else +#define INIT_GETLOADAVG +#endif + +#if SANITIZER_INTERCEPT_MCHECK_MPROBE +INTERCEPTOR(int, mcheck, void (*abortfunc)(int mstatus)) { + return 0; +} + +INTERCEPTOR(int, mcheck_pedantic, void (*abortfunc)(int mstatus)) { + return 0; +} + +INTERCEPTOR(int, mprobe, void *ptr) { + return 0; +} +#endif + +INTERCEPTOR(SIZE_T, wcslen, const wchar_t *s) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, wcslen, s); + SIZE_T res = REAL(wcslen)(s); + COMMON_INTERCEPTOR_READ_RANGE(ctx, s, sizeof(wchar_t) * (res + 1)); + return res; +} + +INTERCEPTOR(SIZE_T, wcsnlen, const wchar_t *s, SIZE_T n) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, wcsnlen, s, n); + SIZE_T res = REAL(wcsnlen)(s, n); + COMMON_INTERCEPTOR_READ_RANGE(ctx, s, sizeof(wchar_t) * Min(res + 1, n)); + return res; +} +#define INIT_WCSLEN \ + COMMON_INTERCEPT_FUNCTION(wcslen); \ + COMMON_INTERCEPT_FUNCTION(wcsnlen); + +#if SANITIZER_INTERCEPT_WCSCAT +INTERCEPTOR(wchar_t *, wcscat, wchar_t *dst, const wchar_t *src) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, wcscat, dst, src); + SIZE_T src_size = REAL(wcslen)(src); + SIZE_T dst_size = REAL(wcslen)(dst); + COMMON_INTERCEPTOR_READ_RANGE(ctx, src, (src_size + 1) * sizeof(wchar_t)); + COMMON_INTERCEPTOR_READ_RANGE(ctx, dst, (dst_size + 1) * sizeof(wchar_t)); + COMMON_INTERCEPTOR_WRITE_RANGE(ctx, dst + dst_size, + (src_size + 1) * sizeof(wchar_t)); + return REAL(wcscat)(dst, src); // NOLINT +} + +INTERCEPTOR(wchar_t *, wcsncat, wchar_t *dst, const wchar_t *src, SIZE_T n) { + void *ctx; + COMMON_INTERCEPTOR_ENTER(ctx, wcsncat, dst, src, n); + SIZE_T src_size = REAL(wcsnlen)(src, n); + SIZE_T dst_size = REAL(wcslen)(dst); + COMMON_INTERCEPTOR_READ_RANGE(ctx, src, + Min(src_size + 1, n) * sizeof(wchar_t)); + COMMON_INTERCEPTOR_READ_RANGE(ctx, dst, (dst_size + 1) * sizeof(wchar_t)); + COMMON_INTERCEPTOR_WRITE_RANGE(ctx, dst + dst_size, + (src_size + 1) * sizeof(wchar_t)); + return REAL(wcsncat)(dst, src, n); // NOLINT +} +#define INIT_WCSCAT \ + COMMON_INTERCEPT_FUNCTION(wcscat); \ + COMMON_INTERCEPT_FUNCTION(wcsncat); +#else +#define INIT_WCSCAT +#endif + static void InitializeCommonInterceptors() { static u64 metadata_mem[sizeof(MetadataHashMap) / sizeof(u64) + 1]; interceptor_metadata_map = new((void *)&metadata_mem) MetadataHashMap(); @@ -6033,6 +6258,8 @@ static void InitializeCommonInterceptors() { INIT_TEXTDOMAIN; INIT_STRLEN; INIT_STRNLEN; + INIT_STRNDUP; + INIT___STRNDUP; INIT_STRCMP; INIT_STRNCMP; INIT_STRCASECMP; @@ -6043,6 +6270,7 @@ static void InitializeCommonInterceptors() { INIT_STRCHRNUL; INIT_STRRCHR; INIT_STRSPN; + INIT_STRTOK; INIT_STRPBRK; INIT_MEMSET; INIT_MEMMOVE; @@ -6052,12 +6280,14 @@ static void InitializeCommonInterceptors() { INIT_MEMRCHR; INIT_MEMMEM; INIT_READ; + INIT_FREAD; INIT_PREAD; INIT_PREAD64; INIT_READV; INIT_PREADV; INIT_PREADV64; INIT_WRITE; + INIT_FWRITE; INIT_PWRITE; INIT_PWRITE64; INIT_WRITEV; @@ -6224,4 +6454,7 @@ static void InitializeCommonInterceptors() { // FIXME: add other *stat interceptors. INIT_UTMP; INIT_UTMPX; + INIT_GETLOADAVG; + INIT_WCSLEN; + INIT_WCSCAT; } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_format.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_format.inc index 1256349..5ebe5a6 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_format.inc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_format.inc @@ -325,8 +325,8 @@ static void scanf_common(void *ctx, int n_inputs, bool allowGnuMalloc, continue; int size = scanf_get_value_size(&dir); if (size == FSS_INVALID) { - Report("WARNING: unexpected format specifier in scanf interceptor: " - "%.*s\n", dir.end - dir.begin, dir.begin); + Report("%s: WARNING: unexpected format specifier in scanf interceptor: ", + SanitizerToolName, "%.*s\n", dir.end - dir.begin, dir.begin); break; } void *argp = va_arg(aq, void *); @@ -520,8 +520,12 @@ static void printf_common(void *ctx, const char *format, va_list aq) { continue; int size = printf_get_value_size(&dir); if (size == FSS_INVALID) { - Report("WARNING: unexpected format specifier in printf " - "interceptor: %.*s\n", dir.end - dir.begin, dir.begin); + static int ReportedOnce; + if (!ReportedOnce++) + Report( + "%s: WARNING: unexpected format specifier in printf " + "interceptor: %.*s (reported once per process)\n", + SanitizerToolName, dir.end - dir.begin, dir.begin); break; } if (dir.convSpecifier == 'n') { diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface.inc new file mode 100644 index 0000000..550427c --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface.inc @@ -0,0 +1,39 @@ +//===-- sanitizer_common_interface.inc ------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// Sanitizer Common interface list. +//===----------------------------------------------------------------------===// +INTERFACE_FUNCTION(__sanitizer_annotate_contiguous_container) +INTERFACE_FUNCTION(__sanitizer_contiguous_container_find_bad_address) +INTERFACE_FUNCTION(__sanitizer_set_death_callback) +INTERFACE_FUNCTION(__sanitizer_set_report_path) +INTERFACE_FUNCTION(__sanitizer_set_report_fd) +INTERFACE_FUNCTION(__sanitizer_verify_contiguous_container) +INTERFACE_WEAK_FUNCTION(__sanitizer_report_error_summary) +INTERFACE_WEAK_FUNCTION(__sanitizer_sandbox_on_notify) +// Sanitizer weak hooks +INTERFACE_WEAK_FUNCTION(__sanitizer_weak_hook_memcmp) +INTERFACE_WEAK_FUNCTION(__sanitizer_weak_hook_strcmp) +INTERFACE_WEAK_FUNCTION(__sanitizer_weak_hook_strncmp) +INTERFACE_WEAK_FUNCTION(__sanitizer_weak_hook_strstr) +// Stacktrace interface. +INTERFACE_FUNCTION(__sanitizer_get_module_and_offset_for_pc) +INTERFACE_FUNCTION(__sanitizer_symbolize_global) +INTERFACE_FUNCTION(__sanitizer_symbolize_pc) +// Allocator interface. +INTERFACE_FUNCTION(__sanitizer_get_allocated_size) +INTERFACE_FUNCTION(__sanitizer_get_current_allocated_bytes) +INTERFACE_FUNCTION(__sanitizer_get_estimated_allocated_size) +INTERFACE_FUNCTION(__sanitizer_get_free_bytes) +INTERFACE_FUNCTION(__sanitizer_get_heap_size) +INTERFACE_FUNCTION(__sanitizer_get_ownership) +INTERFACE_FUNCTION(__sanitizer_get_unmapped_bytes) +INTERFACE_FUNCTION(__sanitizer_install_malloc_and_free_hooks) +INTERFACE_FUNCTION(__sanitizer_print_memory_profile) +INTERFACE_WEAK_FUNCTION(__sanitizer_free_hook) +INTERFACE_WEAK_FUNCTION(__sanitizer_malloc_hook) diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface_posix.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface_posix.inc new file mode 100644 index 0000000..bbc725a --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_interface_posix.inc @@ -0,0 +1,14 @@ +//===-- sanitizer_common_interface_posix.inc ------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// Sanitizer Common interface list only available for Posix systems. +//===----------------------------------------------------------------------===// +INTERFACE_WEAK_FUNCTION(__sanitizer_symbolize_code) +INTERFACE_WEAK_FUNCTION(__sanitizer_symbolize_data) +INTERFACE_WEAK_FUNCTION(__sanitizer_symbolize_demangle) +INTERFACE_WEAK_FUNCTION(__sanitizer_symbolize_flush) diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cc index 49ca961..cf20051 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cc @@ -47,7 +47,8 @@ void SetSandboxingCallback(void (*f)()) { sandboxing_callback = f; } -void ReportErrorSummary(const char *error_type, const StackTrace *stack) { +void ReportErrorSummary(const char *error_type, const StackTrace *stack, + const char *alt_tool_name) { #if !SANITIZER_GO if (!common_flags()->print_summary) return; @@ -59,7 +60,7 @@ void ReportErrorSummary(const char *error_type, const StackTrace *stack) { // Maybe sometimes we need to choose another frame (e.g. skip memcpy/etc). uptr pc = StackTrace::GetPreviousInstructionPc(stack->trace[0]); SymbolizedStack *frame = Symbolizer::GetOrInit()->SymbolizePC(pc); - ReportErrorSummary(error_type, frame->info); + ReportErrorSummary(error_type, frame->info, alt_tool_name); frame->ClearAll(); #endif } @@ -123,7 +124,7 @@ void BackgroundThread(void *arg) { if (heap_profile && current_rss_mb > rss_during_last_reported_profile * 1.1) { Printf("\n\nHEAP PROFILE at RSS %zdMb\n", current_rss_mb); - __sanitizer_print_memory_profile(90); + __sanitizer_print_memory_profile(90, 20); rss_during_last_reported_profile = current_rss_mb; } } @@ -162,8 +163,8 @@ void MaybeStartBackgroudThread() { } // namespace __sanitizer -void NOINLINE -__sanitizer_sandbox_on_notify(__sanitizer_sandbox_arguments *args) { +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_sandbox_on_notify, + __sanitizer_sandbox_arguments *args) { __sanitizer::PrepareForSandboxing(args); if (__sanitizer::sandboxing_callback) __sanitizer::sandboxing_callback(); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_interface.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_interface.inc new file mode 100644 index 0000000..d474900 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_interface.inc @@ -0,0 +1,26 @@ +//===-- sanitizer_coverage_interface.inc ----------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// Sanitizer Coverage interface list. +//===----------------------------------------------------------------------===// +INTERFACE_FUNCTION(__sanitizer_cov_dump) +INTERFACE_FUNCTION(__sanitizer_dump_coverage) +INTERFACE_FUNCTION(__sanitizer_dump_trace_pc_guard_coverage) +INTERFACE_WEAK_FUNCTION(__sancov_default_options) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_cmp) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_cmp1) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_cmp2) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_cmp4) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_cmp8) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_div4) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_div8) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_gep) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_pc_guard) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_pc_guard_init) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_pc_indir) +INTERFACE_WEAK_FUNCTION(__sanitizer_cov_trace_switch) diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc deleted file mode 100644 index 5945ebb..0000000 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc +++ /dev/null @@ -1,1044 +0,0 @@ -//===-- sanitizer_coverage.cc ---------------------------------------------===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// Sanitizer Coverage. -// This file implements run-time support for a poor man's coverage tool. -// -// Compiler instrumentation: -// For every interesting basic block the compiler injects the following code: -// if (Guard < 0) { -// __sanitizer_cov(&Guard); -// } -// At the module start up time __sanitizer_cov_module_init sets the guards -// to consecutive negative numbers (-1, -2, -3, ...). -// It's fine to call __sanitizer_cov more than once for a given block. -// -// Run-time: -// - __sanitizer_cov(): record that we've executed the PC (GET_CALLER_PC). -// and atomically set Guard to -Guard. -// - __sanitizer_cov_dump: dump the coverage data to disk. -// For every module of the current process that has coverage data -// this will create a file module_name.PID.sancov. -// -// The file format is simple: the first 8 bytes is the magic, -// one of 0xC0BFFFFFFFFFFF64 and 0xC0BFFFFFFFFFFF32. The last byte of the -// magic defines the size of the following offsets. -// The rest of the data is the offsets in the module. -// -// Eventually, this coverage implementation should be obsoleted by a more -// powerful general purpose Clang/LLVM coverage instrumentation. -// Consider this implementation as prototype. -// -// FIXME: support (or at least test with) dlclose. -//===----------------------------------------------------------------------===// - -#include "sanitizer_allocator_internal.h" -#include "sanitizer_common.h" -#include "sanitizer_libc.h" -#include "sanitizer_mutex.h" -#include "sanitizer_procmaps.h" -#include "sanitizer_stacktrace.h" -#include "sanitizer_symbolizer.h" -#include "sanitizer_flags.h" - -using namespace __sanitizer; - -static const u64 kMagic64 = 0xC0BFFFFFFFFFFF64ULL; -static const u64 kMagic32 = 0xC0BFFFFFFFFFFF32ULL; -static const uptr kNumWordsForMagic = SANITIZER_WORDSIZE == 64 ? 1 : 2; -static const u64 kMagic = SANITIZER_WORDSIZE == 64 ? kMagic64 : kMagic32; - -static atomic_uint32_t dump_once_guard; // Ensure that CovDump runs only once. - -static atomic_uintptr_t coverage_counter; -static atomic_uintptr_t caller_callee_counter; - -static void ResetGlobalCounters() { - return atomic_store(&coverage_counter, 0, memory_order_relaxed); - return atomic_store(&caller_callee_counter, 0, memory_order_relaxed); -} - -// pc_array is the array containing the covered PCs. -// To make the pc_array thread- and async-signal-safe it has to be large enough. -// 128M counters "ought to be enough for anybody" (4M on 32-bit). - -// With coverage_direct=1 in ASAN_OPTIONS, pc_array memory is mapped to a file. -// In this mode, __sanitizer_cov_dump does nothing, and CovUpdateMapping() -// dump current memory layout to another file. - -static bool cov_sandboxed = false; -static fd_t cov_fd = kInvalidFd; -static unsigned int cov_max_block_size = 0; -static bool coverage_enabled = false; -static const char *coverage_dir; - -namespace __sanitizer { - -class CoverageData { - public: - void Init(); - void Enable(); - void Disable(); - void ReInit(); - void BeforeFork(); - void AfterFork(int child_pid); - void Extend(uptr npcs); - void Add(uptr pc, u32 *guard); - void IndirCall(uptr caller, uptr callee, uptr callee_cache[], - uptr cache_size); - void DumpCallerCalleePairs(); - void DumpTrace(); - void DumpAsBitSet(); - void DumpCounters(); - void DumpOffsets(); - void DumpAll(); - - ALWAYS_INLINE - void TraceBasicBlock(u32 *id); - - void InitializeGuardArray(s32 *guards); - void InitializeGuards(s32 *guards, uptr n, const char *module_name, - uptr caller_pc); - void InitializeCounters(u8 *counters, uptr n); - void ReinitializeGuards(); - uptr GetNumberOf8bitCounters(); - uptr Update8bitCounterBitsetAndClearCounters(u8 *bitset); - - uptr *data(); - uptr size() const; - - private: - struct NamedPcRange { - const char *copied_module_name; - uptr beg, end; // elements [beg,end) in pc_array. - }; - - void DirectOpen(); - void UpdateModuleNameVec(uptr caller_pc, uptr range_beg, uptr range_end); - void GetRangeOffsets(const NamedPcRange& r, Symbolizer* s, - InternalMmapVector<uptr>* offsets) const; - - // Maximal size pc array may ever grow. - // We MmapNoReserve this space to ensure that the array is contiguous. - static const uptr kPcArrayMaxSize = - FIRST_32_SECOND_64(1 << (SANITIZER_ANDROID ? 24 : 26), 1 << 27); - // The amount file mapping for the pc array is grown by. - static const uptr kPcArrayMmapSize = 64 * 1024; - - // pc_array is allocated with MmapNoReserveOrDie and so it uses only as - // much RAM as it really needs. - uptr *pc_array; - // Index of the first available pc_array slot. - atomic_uintptr_t pc_array_index; - // Array size. - atomic_uintptr_t pc_array_size; - // Current file mapped size of the pc array. - uptr pc_array_mapped_size; - // Descriptor of the file mapped pc array. - fd_t pc_fd; - - // Vector of coverage guard arrays, protected by mu. - InternalMmapVectorNoCtor<s32*> guard_array_vec; - - // Vector of module and compilation unit pc ranges. - InternalMmapVectorNoCtor<NamedPcRange> comp_unit_name_vec; - InternalMmapVectorNoCtor<NamedPcRange> module_name_vec; - - struct CounterAndSize { - u8 *counters; - uptr n; - }; - - InternalMmapVectorNoCtor<CounterAndSize> counters_vec; - uptr num_8bit_counters; - - // Caller-Callee (cc) array, size and current index. - static const uptr kCcArrayMaxSize = FIRST_32_SECOND_64(1 << 18, 1 << 24); - uptr **cc_array; - atomic_uintptr_t cc_array_index; - atomic_uintptr_t cc_array_size; - - // Tracing event array, size and current pointer. - // We record all events (basic block entries) in a global buffer of u32 - // values. Each such value is the index in pc_array. - // So far the tracing is highly experimental: - // - not thread-safe; - // - does not support long traces; - // - not tuned for performance. - static const uptr kTrEventArrayMaxSize = FIRST_32_SECOND_64(1 << 22, 1 << 30); - u32 *tr_event_array; - uptr tr_event_array_size; - u32 *tr_event_pointer; - static const uptr kTrPcArrayMaxSize = FIRST_32_SECOND_64(1 << 22, 1 << 27); - - StaticSpinMutex mu; -}; - -static CoverageData coverage_data; - -void CovUpdateMapping(const char *path, uptr caller_pc = 0); - -void CoverageData::DirectOpen() { - InternalScopedString path(kMaxPathLength); - internal_snprintf((char *)path.data(), path.size(), "%s/%zd.sancov.raw", - coverage_dir, internal_getpid()); - pc_fd = OpenFile(path.data(), RdWr); - if (pc_fd == kInvalidFd) { - Report("Coverage: failed to open %s for reading/writing\n", path.data()); - Die(); - } - - pc_array_mapped_size = 0; - CovUpdateMapping(coverage_dir); -} - -void CoverageData::Init() { - pc_fd = kInvalidFd; -} - -void CoverageData::Enable() { - if (pc_array) - return; - pc_array = reinterpret_cast<uptr *>( - MmapNoReserveOrDie(sizeof(uptr) * kPcArrayMaxSize, "CovInit")); - atomic_store(&pc_array_index, 0, memory_order_relaxed); - if (common_flags()->coverage_direct) { - atomic_store(&pc_array_size, 0, memory_order_relaxed); - } else { - atomic_store(&pc_array_size, kPcArrayMaxSize, memory_order_relaxed); - } - - cc_array = reinterpret_cast<uptr **>(MmapNoReserveOrDie( - sizeof(uptr *) * kCcArrayMaxSize, "CovInit::cc_array")); - atomic_store(&cc_array_size, kCcArrayMaxSize, memory_order_relaxed); - atomic_store(&cc_array_index, 0, memory_order_relaxed); - - // Allocate tr_event_array with a guard page at the end. - tr_event_array = reinterpret_cast<u32 *>(MmapNoReserveOrDie( - sizeof(tr_event_array[0]) * kTrEventArrayMaxSize + GetMmapGranularity(), - "CovInit::tr_event_array")); - MprotectNoAccess( - reinterpret_cast<uptr>(&tr_event_array[kTrEventArrayMaxSize]), - GetMmapGranularity()); - tr_event_array_size = kTrEventArrayMaxSize; - tr_event_pointer = tr_event_array; - - num_8bit_counters = 0; -} - -void CoverageData::InitializeGuardArray(s32 *guards) { - Enable(); // Make sure coverage is enabled at this point. - s32 n = guards[0]; - for (s32 j = 1; j <= n; j++) { - uptr idx = atomic_load_relaxed(&pc_array_index); - atomic_store_relaxed(&pc_array_index, idx + 1); - guards[j] = -static_cast<s32>(idx + 1); - } -} - -void CoverageData::Disable() { - if (pc_array) { - UnmapOrDie(pc_array, sizeof(uptr) * kPcArrayMaxSize); - pc_array = nullptr; - } - if (cc_array) { - UnmapOrDie(cc_array, sizeof(uptr *) * kCcArrayMaxSize); - cc_array = nullptr; - } - if (tr_event_array) { - UnmapOrDie(tr_event_array, - sizeof(tr_event_array[0]) * kTrEventArrayMaxSize + - GetMmapGranularity()); - tr_event_array = nullptr; - tr_event_pointer = nullptr; - } - if (pc_fd != kInvalidFd) { - CloseFile(pc_fd); - pc_fd = kInvalidFd; - } -} - -void CoverageData::ReinitializeGuards() { - // Assuming single thread. - atomic_store(&pc_array_index, 0, memory_order_relaxed); - for (uptr i = 0; i < guard_array_vec.size(); i++) - InitializeGuardArray(guard_array_vec[i]); -} - -void CoverageData::ReInit() { - Disable(); - if (coverage_enabled) { - if (common_flags()->coverage_direct) { - // In memory-mapped mode we must extend the new file to the known array - // size. - uptr size = atomic_load(&pc_array_size, memory_order_relaxed); - uptr npcs = size / sizeof(uptr); - Enable(); - if (size) Extend(npcs); - if (coverage_enabled) CovUpdateMapping(coverage_dir); - } else { - Enable(); - } - } - // Re-initialize the guards. - // We are single-threaded now, no need to grab any lock. - CHECK_EQ(atomic_load(&pc_array_index, memory_order_relaxed), 0); - ReinitializeGuards(); -} - -void CoverageData::BeforeFork() { - mu.Lock(); -} - -void CoverageData::AfterFork(int child_pid) { - // We are single-threaded so it's OK to release the lock early. - mu.Unlock(); - if (child_pid == 0) ReInit(); -} - -// Extend coverage PC array to fit additional npcs elements. -void CoverageData::Extend(uptr npcs) { - if (!common_flags()->coverage_direct) return; - SpinMutexLock l(&mu); - - uptr size = atomic_load(&pc_array_size, memory_order_relaxed); - size += npcs * sizeof(uptr); - - if (coverage_enabled && size > pc_array_mapped_size) { - if (pc_fd == kInvalidFd) DirectOpen(); - CHECK_NE(pc_fd, kInvalidFd); - - uptr new_mapped_size = pc_array_mapped_size; - while (size > new_mapped_size) new_mapped_size += kPcArrayMmapSize; - CHECK_LE(new_mapped_size, sizeof(uptr) * kPcArrayMaxSize); - - // Extend the file and map the new space at the end of pc_array. - uptr res = internal_ftruncate(pc_fd, new_mapped_size); - int err; - if (internal_iserror(res, &err)) { - Printf("failed to extend raw coverage file: %d\n", err); - Die(); - } - - uptr next_map_base = ((uptr)pc_array) + pc_array_mapped_size; - void *p = MapWritableFileToMemory((void *)next_map_base, - new_mapped_size - pc_array_mapped_size, - pc_fd, pc_array_mapped_size); - CHECK_EQ((uptr)p, next_map_base); - pc_array_mapped_size = new_mapped_size; - } - - atomic_store(&pc_array_size, size, memory_order_release); -} - -void CoverageData::InitializeCounters(u8 *counters, uptr n) { - if (!counters) return; - CHECK_EQ(reinterpret_cast<uptr>(counters) % 16, 0); - n = RoundUpTo(n, 16); // The compiler must ensure that counters is 16-aligned. - SpinMutexLock l(&mu); - counters_vec.push_back({counters, n}); - num_8bit_counters += n; -} - -void CoverageData::UpdateModuleNameVec(uptr caller_pc, uptr range_beg, - uptr range_end) { - auto sym = Symbolizer::GetOrInit(); - if (!sym) - return; - const char *module_name = sym->GetModuleNameForPc(caller_pc); - if (!module_name) return; - if (module_name_vec.empty() || - module_name_vec.back().copied_module_name != module_name) - module_name_vec.push_back({module_name, range_beg, range_end}); - else - module_name_vec.back().end = range_end; -} - -void CoverageData::InitializeGuards(s32 *guards, uptr n, - const char *comp_unit_name, - uptr caller_pc) { - // The array 'guards' has n+1 elements, we use the element zero - // to store 'n'. - CHECK_LT(n, 1 << 30); - guards[0] = static_cast<s32>(n); - InitializeGuardArray(guards); - SpinMutexLock l(&mu); - uptr range_end = atomic_load(&pc_array_index, memory_order_relaxed); - uptr range_beg = range_end - n; - comp_unit_name_vec.push_back({comp_unit_name, range_beg, range_end}); - guard_array_vec.push_back(guards); - UpdateModuleNameVec(caller_pc, range_beg, range_end); -} - -static const uptr kBundleCounterBits = 16; - -// When coverage_order_pcs==true and SANITIZER_WORDSIZE==64 -// we insert the global counter into the first 16 bits of the PC. -uptr BundlePcAndCounter(uptr pc, uptr counter) { - if (SANITIZER_WORDSIZE != 64 || !common_flags()->coverage_order_pcs) - return pc; - static const uptr kMaxCounter = (1 << kBundleCounterBits) - 1; - if (counter > kMaxCounter) - counter = kMaxCounter; - CHECK_EQ(0, pc >> (SANITIZER_WORDSIZE - kBundleCounterBits)); - return pc | (counter << (SANITIZER_WORDSIZE - kBundleCounterBits)); -} - -uptr UnbundlePc(uptr bundle) { - if (SANITIZER_WORDSIZE != 64 || !common_flags()->coverage_order_pcs) - return bundle; - return (bundle << kBundleCounterBits) >> kBundleCounterBits; -} - -uptr UnbundleCounter(uptr bundle) { - if (SANITIZER_WORDSIZE != 64 || !common_flags()->coverage_order_pcs) - return 0; - return bundle >> (SANITIZER_WORDSIZE - kBundleCounterBits); -} - -// If guard is negative, atomically set it to -guard and store the PC in -// pc_array. -void CoverageData::Add(uptr pc, u32 *guard) { - atomic_uint32_t *atomic_guard = reinterpret_cast<atomic_uint32_t*>(guard); - s32 guard_value = atomic_load(atomic_guard, memory_order_relaxed); - if (guard_value >= 0) return; - - atomic_store(atomic_guard, -guard_value, memory_order_relaxed); - if (!pc_array) return; - - uptr idx = -guard_value - 1; - if (idx >= atomic_load(&pc_array_index, memory_order_acquire)) - return; // May happen after fork when pc_array_index becomes 0. - CHECK_LT(idx * sizeof(uptr), - atomic_load(&pc_array_size, memory_order_acquire)); - uptr counter = atomic_fetch_add(&coverage_counter, 1, memory_order_relaxed); - pc_array[idx] = BundlePcAndCounter(pc, counter); -} - -// Registers a pair caller=>callee. -// When a given caller is seen for the first time, the callee_cache is added -// to the global array cc_array, callee_cache[0] is set to caller and -// callee_cache[1] is set to cache_size. -// Then we are trying to add callee to callee_cache [2,cache_size) if it is -// not there yet. -// If the cache is full we drop the callee (may want to fix this later). -void CoverageData::IndirCall(uptr caller, uptr callee, uptr callee_cache[], - uptr cache_size) { - if (!cc_array) return; - atomic_uintptr_t *atomic_callee_cache = - reinterpret_cast<atomic_uintptr_t *>(callee_cache); - uptr zero = 0; - if (atomic_compare_exchange_strong(&atomic_callee_cache[0], &zero, caller, - memory_order_seq_cst)) { - uptr idx = atomic_fetch_add(&cc_array_index, 1, memory_order_relaxed); - CHECK_LT(idx * sizeof(uptr), - atomic_load(&cc_array_size, memory_order_acquire)); - callee_cache[1] = cache_size; - cc_array[idx] = callee_cache; - } - CHECK_EQ(atomic_load(&atomic_callee_cache[0], memory_order_relaxed), caller); - for (uptr i = 2; i < cache_size; i++) { - uptr was = 0; - if (atomic_compare_exchange_strong(&atomic_callee_cache[i], &was, callee, - memory_order_seq_cst)) { - atomic_fetch_add(&caller_callee_counter, 1, memory_order_relaxed); - return; - } - if (was == callee) // Already have this callee. - return; - } -} - -uptr CoverageData::GetNumberOf8bitCounters() { - return num_8bit_counters; -} - -// Map every 8bit counter to a 8-bit bitset and clear the counter. -uptr CoverageData::Update8bitCounterBitsetAndClearCounters(u8 *bitset) { - uptr num_new_bits = 0; - uptr cur = 0; - // For better speed we map 8 counters to 8 bytes of bitset at once. - static const uptr kBatchSize = 8; - CHECK_EQ(reinterpret_cast<uptr>(bitset) % kBatchSize, 0); - for (uptr i = 0, len = counters_vec.size(); i < len; i++) { - u8 *c = counters_vec[i].counters; - uptr n = counters_vec[i].n; - CHECK_EQ(n % 16, 0); - CHECK_EQ(cur % kBatchSize, 0); - CHECK_EQ(reinterpret_cast<uptr>(c) % kBatchSize, 0); - if (!bitset) { - internal_bzero_aligned16(c, n); - cur += n; - continue; - } - for (uptr j = 0; j < n; j += kBatchSize, cur += kBatchSize) { - CHECK_LT(cur, num_8bit_counters); - u64 *pc64 = reinterpret_cast<u64*>(c + j); - u64 *pb64 = reinterpret_cast<u64*>(bitset + cur); - u64 c64 = *pc64; - u64 old_bits_64 = *pb64; - u64 new_bits_64 = old_bits_64; - if (c64) { - *pc64 = 0; - for (uptr k = 0; k < kBatchSize; k++) { - u64 x = (c64 >> (8 * k)) & 0xff; - if (x) { - u64 bit = 0; - /**/ if (x >= 128) bit = 128; - else if (x >= 32) bit = 64; - else if (x >= 16) bit = 32; - else if (x >= 8) bit = 16; - else if (x >= 4) bit = 8; - else if (x >= 3) bit = 4; - else if (x >= 2) bit = 2; - else if (x >= 1) bit = 1; - u64 mask = bit << (8 * k); - if (!(new_bits_64 & mask)) { - num_new_bits++; - new_bits_64 |= mask; - } - } - } - *pb64 = new_bits_64; - } - } - } - CHECK_EQ(cur, num_8bit_counters); - return num_new_bits; -} - -uptr *CoverageData::data() { - return pc_array; -} - -uptr CoverageData::size() const { - return atomic_load(&pc_array_index, memory_order_relaxed); -} - -// Block layout for packed file format: header, followed by module name (no -// trailing zero), followed by data blob. -struct CovHeader { - int pid; - unsigned int module_name_length; - unsigned int data_length; -}; - -static void CovWritePacked(int pid, const char *module, const void *blob, - unsigned int blob_size) { - if (cov_fd == kInvalidFd) return; - unsigned module_name_length = internal_strlen(module); - CovHeader header = {pid, module_name_length, blob_size}; - - if (cov_max_block_size == 0) { - // Writing to a file. Just go ahead. - WriteToFile(cov_fd, &header, sizeof(header)); - WriteToFile(cov_fd, module, module_name_length); - WriteToFile(cov_fd, blob, blob_size); - } else { - // Writing to a socket. We want to split the data into appropriately sized - // blocks. - InternalScopedBuffer<char> block(cov_max_block_size); - CHECK_EQ((uptr)block.data(), (uptr)(CovHeader *)block.data()); - uptr header_size_with_module = sizeof(header) + module_name_length; - CHECK_LT(header_size_with_module, cov_max_block_size); - unsigned int max_payload_size = - cov_max_block_size - header_size_with_module; - char *block_pos = block.data(); - internal_memcpy(block_pos, &header, sizeof(header)); - block_pos += sizeof(header); - internal_memcpy(block_pos, module, module_name_length); - block_pos += module_name_length; - char *block_data_begin = block_pos; - const char *blob_pos = (const char *)blob; - while (blob_size > 0) { - unsigned int payload_size = Min(blob_size, max_payload_size); - blob_size -= payload_size; - internal_memcpy(block_data_begin, blob_pos, payload_size); - blob_pos += payload_size; - ((CovHeader *)block.data())->data_length = payload_size; - WriteToFile(cov_fd, block.data(), header_size_with_module + payload_size); - } - } -} - -// If packed = false: <name>.<pid>.<sancov> (name = module name). -// If packed = true and name == 0: <pid>.<sancov>.<packed>. -// If packed = true and name != 0: <name>.<sancov>.<packed> (name is -// user-supplied). -static fd_t CovOpenFile(InternalScopedString *path, bool packed, - const char *name, const char *extension = "sancov") { - path->clear(); - if (!packed) { - CHECK(name); - path->append("%s/%s.%zd.%s", coverage_dir, name, internal_getpid(), - extension); - } else { - if (!name) - path->append("%s/%zd.%s.packed", coverage_dir, internal_getpid(), - extension); - else - path->append("%s/%s.%s.packed", coverage_dir, name, extension); - } - error_t err; - fd_t fd = OpenFile(path->data(), WrOnly, &err); - if (fd == kInvalidFd) - Report("SanitizerCoverage: failed to open %s for writing (reason: %d)\n", - path->data(), err); - return fd; -} - -// Dump trace PCs and trace events into two separate files. -void CoverageData::DumpTrace() { - uptr max_idx = tr_event_pointer - tr_event_array; - if (!max_idx) return; - auto sym = Symbolizer::GetOrInit(); - if (!sym) - return; - InternalScopedString out(32 << 20); - for (uptr i = 0, n = size(); i < n; i++) { - const char *module_name = "<unknown>"; - uptr module_address = 0; - sym->GetModuleNameAndOffsetForPC(UnbundlePc(pc_array[i]), &module_name, - &module_address); - out.append("%s 0x%zx\n", module_name, module_address); - } - InternalScopedString path(kMaxPathLength); - fd_t fd = CovOpenFile(&path, false, "trace-points"); - if (fd == kInvalidFd) return; - WriteToFile(fd, out.data(), out.length()); - CloseFile(fd); - - fd = CovOpenFile(&path, false, "trace-compunits"); - if (fd == kInvalidFd) return; - out.clear(); - for (uptr i = 0; i < comp_unit_name_vec.size(); i++) - out.append("%s\n", comp_unit_name_vec[i].copied_module_name); - WriteToFile(fd, out.data(), out.length()); - CloseFile(fd); - - fd = CovOpenFile(&path, false, "trace-events"); - if (fd == kInvalidFd) return; - uptr bytes_to_write = max_idx * sizeof(tr_event_array[0]); - u8 *event_bytes = reinterpret_cast<u8*>(tr_event_array); - // The trace file could be huge, and may not be written with a single syscall. - while (bytes_to_write) { - uptr actually_written; - if (WriteToFile(fd, event_bytes, bytes_to_write, &actually_written) && - actually_written <= bytes_to_write) { - bytes_to_write -= actually_written; - event_bytes += actually_written; - } else { - break; - } - } - CloseFile(fd); - VReport(1, " CovDump: Trace: %zd PCs written\n", size()); - VReport(1, " CovDump: Trace: %zd Events written\n", max_idx); -} - -// This function dumps the caller=>callee pairs into a file as a sequence of -// lines like "module_name offset". -void CoverageData::DumpCallerCalleePairs() { - uptr max_idx = atomic_load(&cc_array_index, memory_order_relaxed); - if (!max_idx) return; - auto sym = Symbolizer::GetOrInit(); - if (!sym) - return; - InternalScopedString out(32 << 20); - uptr total = 0; - for (uptr i = 0; i < max_idx; i++) { - uptr *cc_cache = cc_array[i]; - CHECK(cc_cache); - uptr caller = cc_cache[0]; - uptr n_callees = cc_cache[1]; - const char *caller_module_name = "<unknown>"; - uptr caller_module_address = 0; - sym->GetModuleNameAndOffsetForPC(caller, &caller_module_name, - &caller_module_address); - for (uptr j = 2; j < n_callees; j++) { - uptr callee = cc_cache[j]; - if (!callee) break; - total++; - const char *callee_module_name = "<unknown>"; - uptr callee_module_address = 0; - sym->GetModuleNameAndOffsetForPC(callee, &callee_module_name, - &callee_module_address); - out.append("%s 0x%zx\n%s 0x%zx\n", caller_module_name, - caller_module_address, callee_module_name, - callee_module_address); - } - } - InternalScopedString path(kMaxPathLength); - fd_t fd = CovOpenFile(&path, false, "caller-callee"); - if (fd == kInvalidFd) return; - WriteToFile(fd, out.data(), out.length()); - CloseFile(fd); - VReport(1, " CovDump: %zd caller-callee pairs written\n", total); -} - -// Record the current PC into the event buffer. -// Every event is a u32 value (index in tr_pc_array_index) so we compute -// it once and then cache in the provided 'cache' storage. -// -// This function will eventually be inlined by the compiler. -void CoverageData::TraceBasicBlock(u32 *id) { - // Will trap here if - // 1. coverage is not enabled at run-time. - // 2. The array tr_event_array is full. - *tr_event_pointer = *id - 1; - tr_event_pointer++; -} - -void CoverageData::DumpCounters() { - if (!common_flags()->coverage_counters) return; - uptr n = coverage_data.GetNumberOf8bitCounters(); - if (!n) return; - InternalScopedBuffer<u8> bitset(n); - coverage_data.Update8bitCounterBitsetAndClearCounters(bitset.data()); - InternalScopedString path(kMaxPathLength); - - for (uptr m = 0; m < module_name_vec.size(); m++) { - auto r = module_name_vec[m]; - CHECK(r.copied_module_name); - CHECK_LE(r.beg, r.end); - CHECK_LE(r.end, size()); - const char *base_name = StripModuleName(r.copied_module_name); - fd_t fd = - CovOpenFile(&path, /* packed */ false, base_name, "counters-sancov"); - if (fd == kInvalidFd) return; - WriteToFile(fd, bitset.data() + r.beg, r.end - r.beg); - CloseFile(fd); - VReport(1, " CovDump: %zd counters written for '%s'\n", r.end - r.beg, - base_name); - } -} - -void CoverageData::DumpAsBitSet() { - if (!common_flags()->coverage_bitset) return; - if (!size()) return; - InternalScopedBuffer<char> out(size()); - InternalScopedString path(kMaxPathLength); - for (uptr m = 0; m < module_name_vec.size(); m++) { - uptr n_set_bits = 0; - auto r = module_name_vec[m]; - CHECK(r.copied_module_name); - CHECK_LE(r.beg, r.end); - CHECK_LE(r.end, size()); - for (uptr i = r.beg; i < r.end; i++) { - uptr pc = UnbundlePc(pc_array[i]); - out[i] = pc ? '1' : '0'; - if (pc) - n_set_bits++; - } - const char *base_name = StripModuleName(r.copied_module_name); - fd_t fd = CovOpenFile(&path, /* packed */false, base_name, "bitset-sancov"); - if (fd == kInvalidFd) return; - WriteToFile(fd, out.data() + r.beg, r.end - r.beg); - CloseFile(fd); - VReport(1, - " CovDump: bitset of %zd bits written for '%s', %zd bits are set\n", - r.end - r.beg, base_name, n_set_bits); - } -} - - -void CoverageData::GetRangeOffsets(const NamedPcRange& r, Symbolizer* sym, - InternalMmapVector<uptr>* offsets) const { - offsets->clear(); - for (uptr i = 0; i < kNumWordsForMagic; i++) - offsets->push_back(0); - CHECK(r.copied_module_name); - CHECK_LE(r.beg, r.end); - CHECK_LE(r.end, size()); - for (uptr i = r.beg; i < r.end; i++) { - uptr pc = UnbundlePc(pc_array[i]); - uptr counter = UnbundleCounter(pc_array[i]); - if (!pc) continue; // Not visited. - uptr offset = 0; - sym->GetModuleNameAndOffsetForPC(pc, nullptr, &offset); - offsets->push_back(BundlePcAndCounter(offset, counter)); - } - - CHECK_GE(offsets->size(), kNumWordsForMagic); - SortArray(offsets->data(), offsets->size()); - for (uptr i = 0; i < offsets->size(); i++) - (*offsets)[i] = UnbundlePc((*offsets)[i]); -} - -static void GenerateHtmlReport(const InternalMmapVector<char *> &cov_files) { - if (!common_flags()->html_cov_report) { - return; - } - char *sancov_path = FindPathToBinary(common_flags()->sancov_path); - if (sancov_path == nullptr) { - return; - } - - InternalMmapVector<char *> sancov_argv(cov_files.size() * 2 + 3); - sancov_argv.push_back(sancov_path); - sancov_argv.push_back(internal_strdup("-html-report")); - auto argv_deleter = at_scope_exit([&] { - for (uptr i = 0; i < sancov_argv.size(); ++i) { - InternalFree(sancov_argv[i]); - } - }); - - for (const auto &cov_file : cov_files) { - sancov_argv.push_back(internal_strdup(cov_file)); - } - - { - ListOfModules modules; - modules.init(); - for (const LoadedModule &module : modules) { - sancov_argv.push_back(internal_strdup(module.full_name())); - } - } - - InternalScopedString report_path(kMaxPathLength); - fd_t report_fd = - CovOpenFile(&report_path, false /* packed */, GetProcessName(), "html"); - int pid = StartSubprocess(sancov_argv[0], sancov_argv.data(), - kInvalidFd /* stdin */, report_fd /* std_out */); - if (pid > 0) { - int result = WaitForProcess(pid); - if (result == 0) - Printf("coverage report generated to %s\n", report_path.data()); - } -} - -void CoverageData::DumpOffsets() { - auto sym = Symbolizer::GetOrInit(); - if (!common_flags()->coverage_pcs) return; - CHECK_NE(sym, nullptr); - InternalMmapVector<uptr> offsets(0); - InternalScopedString path(kMaxPathLength); - - InternalMmapVector<char *> cov_files(module_name_vec.size()); - auto cov_files_deleter = at_scope_exit([&] { - for (uptr i = 0; i < cov_files.size(); ++i) { - InternalFree(cov_files[i]); - } - }); - - for (uptr m = 0; m < module_name_vec.size(); m++) { - auto r = module_name_vec[m]; - GetRangeOffsets(r, sym, &offsets); - - uptr num_offsets = offsets.size() - kNumWordsForMagic; - u64 *magic_p = reinterpret_cast<u64*>(offsets.data()); - CHECK_EQ(*magic_p, 0ULL); - // FIXME: we may want to write 32-bit offsets even in 64-mode - // if all the offsets are small enough. - *magic_p = kMagic; - - const char *module_name = StripModuleName(r.copied_module_name); - if (cov_sandboxed) { - if (cov_fd != kInvalidFd) { - CovWritePacked(internal_getpid(), module_name, offsets.data(), - offsets.size() * sizeof(offsets[0])); - VReport(1, " CovDump: %zd PCs written to packed file\n", num_offsets); - } - } else { - // One file per module per process. - fd_t fd = CovOpenFile(&path, false /* packed */, module_name); - if (fd == kInvalidFd) continue; - WriteToFile(fd, offsets.data(), offsets.size() * sizeof(offsets[0])); - CloseFile(fd); - cov_files.push_back(internal_strdup(path.data())); - VReport(1, " CovDump: %s: %zd PCs written\n", path.data(), num_offsets); - } - } - if (cov_fd != kInvalidFd) - CloseFile(cov_fd); - - GenerateHtmlReport(cov_files); -} - -void CoverageData::DumpAll() { - if (!coverage_enabled || common_flags()->coverage_direct) return; - if (atomic_fetch_add(&dump_once_guard, 1, memory_order_relaxed)) - return; - DumpAsBitSet(); - DumpCounters(); - DumpTrace(); - DumpOffsets(); - DumpCallerCalleePairs(); -} - -void CovPrepareForSandboxing(__sanitizer_sandbox_arguments *args) { - if (!args) return; - if (!coverage_enabled) return; - cov_sandboxed = args->coverage_sandboxed; - if (!cov_sandboxed) return; - cov_max_block_size = args->coverage_max_block_size; - if (args->coverage_fd >= 0) { - cov_fd = (fd_t)args->coverage_fd; - } else { - InternalScopedString path(kMaxPathLength); - // Pre-open the file now. The sandbox won't allow us to do it later. - cov_fd = CovOpenFile(&path, true /* packed */, nullptr); - } -} - -fd_t MaybeOpenCovFile(const char *name) { - CHECK(name); - if (!coverage_enabled) return kInvalidFd; - InternalScopedString path(kMaxPathLength); - return CovOpenFile(&path, true /* packed */, name); -} - -void CovBeforeFork() { - coverage_data.BeforeFork(); -} - -void CovAfterFork(int child_pid) { - coverage_data.AfterFork(child_pid); -} - -static void MaybeDumpCoverage() { - if (common_flags()->coverage) - __sanitizer_cov_dump(); -} - -void InitializeCoverage(bool enabled, const char *dir) { - if (coverage_enabled) - return; // May happen if two sanitizer enable coverage in the same process. - coverage_enabled = enabled; - coverage_dir = dir; - coverage_data.Init(); - if (enabled) coverage_data.Enable(); - if (!common_flags()->coverage_direct) Atexit(__sanitizer_cov_dump); - AddDieCallback(MaybeDumpCoverage); -} - -void ReInitializeCoverage(bool enabled, const char *dir) { - coverage_enabled = enabled; - coverage_dir = dir; - coverage_data.ReInit(); -} - -void CoverageUpdateMapping() { - if (coverage_enabled) - CovUpdateMapping(coverage_dir); -} - -} // namespace __sanitizer - -extern "C" { -SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov(u32 *guard) { - coverage_data.Add(StackTrace::GetPreviousInstructionPc(GET_CALLER_PC()), - guard); -} -SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_with_check(u32 *guard) { - atomic_uint32_t *atomic_guard = reinterpret_cast<atomic_uint32_t*>(guard); - if (static_cast<s32>( - __sanitizer::atomic_load(atomic_guard, memory_order_relaxed)) < 0) - __sanitizer_cov(guard); -} -SANITIZER_INTERFACE_ATTRIBUTE void -__sanitizer_cov_indir_call16(uptr callee, uptr callee_cache16[]) { - coverage_data.IndirCall(StackTrace::GetPreviousInstructionPc(GET_CALLER_PC()), - callee, callee_cache16, 16); -} -SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_init() { - coverage_enabled = true; - coverage_dir = common_flags()->coverage_dir; - coverage_data.Init(); -} -SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_dump() { - coverage_data.DumpAll(); -#if SANITIZER_LINUX - __sanitizer_dump_trace_pc_guard_coverage(); -#endif -} -SANITIZER_INTERFACE_ATTRIBUTE void -__sanitizer_cov_module_init(s32 *guards, uptr npcs, u8 *counters, - const char *comp_unit_name) { - coverage_data.InitializeGuards(guards, npcs, comp_unit_name, GET_CALLER_PC()); - coverage_data.InitializeCounters(counters, npcs); - if (!common_flags()->coverage_direct) return; - if (SANITIZER_ANDROID && coverage_enabled) { - // dlopen/dlclose interceptors do not work on Android, so we rely on - // Extend() calls to update .sancov.map. - CovUpdateMapping(coverage_dir, GET_CALLER_PC()); - } - coverage_data.Extend(npcs); -} -SANITIZER_INTERFACE_ATTRIBUTE -sptr __sanitizer_maybe_open_cov_file(const char *name) { - return (sptr)MaybeOpenCovFile(name); -} -SANITIZER_INTERFACE_ATTRIBUTE -uptr __sanitizer_get_total_unique_coverage() { - return atomic_load(&coverage_counter, memory_order_relaxed); -} - -SANITIZER_INTERFACE_ATTRIBUTE -uptr __sanitizer_get_total_unique_caller_callee_pairs() { - return atomic_load(&caller_callee_counter, memory_order_relaxed); -} - -SANITIZER_INTERFACE_ATTRIBUTE -void __sanitizer_cov_trace_func_enter(u32 *id) { - __sanitizer_cov_with_check(id); - coverage_data.TraceBasicBlock(id); -} -SANITIZER_INTERFACE_ATTRIBUTE -void __sanitizer_cov_trace_basic_block(u32 *id) { - __sanitizer_cov_with_check(id); - coverage_data.TraceBasicBlock(id); -} -SANITIZER_INTERFACE_ATTRIBUTE -void __sanitizer_reset_coverage() { - ResetGlobalCounters(); - coverage_data.ReinitializeGuards(); - internal_bzero_aligned16( - coverage_data.data(), - RoundUpTo(coverage_data.size() * sizeof(coverage_data.data()[0]), 16)); -} -SANITIZER_INTERFACE_ATTRIBUTE -uptr __sanitizer_get_coverage_guards(uptr **data) { - *data = coverage_data.data(); - return coverage_data.size(); -} - -SANITIZER_INTERFACE_ATTRIBUTE -uptr __sanitizer_get_number_of_counters() { - return coverage_data.GetNumberOf8bitCounters(); -} - -SANITIZER_INTERFACE_ATTRIBUTE -uptr __sanitizer_update_counter_bitset_and_clear_counters(u8 *bitset) { - return coverage_data.Update8bitCounterBitsetAndClearCounters(bitset); -} -// Default empty implementations (weak). Users should redefine them. -#if !SANITIZER_WINDOWS // weak does not work on Windows. -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_cmp() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_cmp1() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_cmp2() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_cmp4() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_cmp8() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_switch() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_div4() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_div8() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_gep() {} -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void __sanitizer_cov_trace_pc_indir() {} -#endif // !SANITIZER_WINDOWS -} // extern "C" diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep_new.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep_new.cc index df6d10f..2443335 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep_new.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep_new.cc @@ -49,7 +49,7 @@ static void WriteModuleCoverage(char* file_path, const char* module_name, WriteToFile(fd, &Magic, sizeof(Magic)); WriteToFile(fd, pcs, len * sizeof(*pcs)); CloseFile(fd); - Printf("SanitizerCoverage: %s %zd PCs written\n", file_path, len); + Printf("SanitizerCoverage: %s: %zd PCs written\n", file_path, len); } static void SanitizerDumpCoverage(const uptr* unsorted_pcs, uptr len) { @@ -71,7 +71,7 @@ static void SanitizerDumpCoverage(const uptr* unsorted_pcs, uptr len) { if (!pc) continue; if (!__sanitizer_get_module_and_offset_for_pc(pc, nullptr, 0, &pcs[i])) { - Printf("ERROR: bad pc %x\n", pc); + Printf("ERROR: unknown pc 0x%x (may happen if dlclose is used)\n", pc); continue; } uptr module_base = pc - pcs[i]; @@ -98,10 +98,6 @@ static void SanitizerDumpCoverage(const uptr* unsorted_pcs, uptr len) { InternalFree(file_path); InternalFree(module_name); InternalFree(pcs); - - if (sancov_flags()->symbolize) { - Printf("TODO(aizatsky): call sancov to symbolize\n"); - } } // Collects trace-pc guard coverage. @@ -150,20 +146,30 @@ static TracePcGuardController pc_guard_controller; } // namespace } // namespace __sancov +namespace __sanitizer { +void InitializeCoverage(bool enabled, const char *dir) { + static bool coverage_enabled = false; + if (coverage_enabled) + return; // May happen if two sanitizer enable coverage in the same process. + coverage_enabled = enabled; + Atexit(__sanitizer_cov_dump); + AddDieCallback(__sanitizer_cov_dump); +} +} // namespace __sanitizer + extern "C" { SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_dump_coverage( // NOLINT const uptr* pcs, uptr len) { return __sancov::SanitizerDumpCoverage(pcs, len); } -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void -__sanitizer_cov_trace_pc_guard(u32* guard) { +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_pc_guard, u32* guard) { if (!*guard) return; __sancov::pc_guard_controller.TracePcGuard(guard, GET_CALLER_PC() - 1); } -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void -__sanitizer_cov_trace_pc_guard_init(u32* start, u32* end) { +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_pc_guard_init, + u32* start, u32* end) { if (start == end || *start) return; __sancov::pc_guard_controller.InitTracePcGuard(start, end); } @@ -171,4 +177,18 @@ __sanitizer_cov_trace_pc_guard_init(u32* start, u32* end) { SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_dump_trace_pc_guard_coverage() { __sancov::pc_guard_controller.Dump(); } +SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_dump() { + __sanitizer_dump_trace_pc_guard_coverage(); +} +// Default empty implementations (weak). Users should redefine them. +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_cmp, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_cmp1, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_cmp2, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_cmp4, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_cmp8, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_switch, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_div4, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_div8, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_gep, void) {} +SANITIZER_INTERFACE_WEAK_DEF(void, __sanitizer_cov_trace_pc_indir, void) {} } // extern "C" diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_mapping_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_mapping_libcdep.cc deleted file mode 100644 index 3477b06..0000000 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_mapping_libcdep.cc +++ /dev/null @@ -1,122 +0,0 @@ -//===-- sanitizer_coverage_mapping.cc -------------------------------------===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// Mmap-based implementation of sanitizer coverage. -// -// This is part of the implementation of code coverage that does not require -// __sanitizer_cov_dump() call. Data is stored in 2 files per process. -// -// $pid.sancov.map describes process memory layout in the following text-based -// format: -// <pointer size in bits> // 1 line, 32 or 64 -// <mapping start> <mapping end> <base address> <dso name> // repeated -// ... -// Mapping lines are NOT sorted. This file is updated every time memory layout -// is changed (i.e. in dlopen() and dlclose() interceptors). -// -// $pid.sancov.raw is a binary dump of PC values, sizeof(uptr) each. Again, not -// sorted. This file is extended by 64Kb at a time and mapped into memory. It -// contains one or more 0 words at the end, up to the next 64Kb aligned offset. -// -// To convert these 2 files to the usual .sancov format, run sancov.py rawunpack -// $pid.sancov.raw. -// -//===----------------------------------------------------------------------===// - -#include "sanitizer_allocator_internal.h" -#include "sanitizer_libc.h" -#include "sanitizer_procmaps.h" - -namespace __sanitizer { - -static const uptr kMaxTextSize = 64 * 1024; - -struct CachedMapping { - public: - bool NeedsUpdate(uptr pc) { - int new_pid = internal_getpid(); - if (last_pid == new_pid && pc && pc >= last_range_start && - pc < last_range_end) - return false; - last_pid = new_pid; - return true; - } - - void SetModuleRange(uptr start, uptr end) { - last_range_start = start; - last_range_end = end; - } - - private: - uptr last_range_start, last_range_end; - int last_pid; -}; - -static CachedMapping cached_mapping; -static StaticSpinMutex mapping_mu; - -void CovUpdateMapping(const char *coverage_dir, uptr caller_pc) { - if (!common_flags()->coverage_direct) return; - - SpinMutexLock l(&mapping_mu); - - if (!cached_mapping.NeedsUpdate(caller_pc)) - return; - - InternalScopedString text(kMaxTextSize); - - { - text.append("%d\n", sizeof(uptr) * 8); - ListOfModules modules; - modules.init(); - for (const LoadedModule &module : modules) { - const char *module_name = StripModuleName(module.full_name()); - uptr base = module.base_address(); - for (const auto &range : module.ranges()) { - if (range.executable) { - uptr start = range.beg; - uptr end = range.end; - text.append("%zx %zx %zx %s\n", start, end, base, module_name); - if (caller_pc && caller_pc >= start && caller_pc < end) - cached_mapping.SetModuleRange(start, end); - } - } - } - } - - error_t err; - InternalScopedString tmp_path(64 + internal_strlen(coverage_dir)); - uptr res = internal_snprintf((char *)tmp_path.data(), tmp_path.size(), - "%s/%zd.sancov.map.tmp", coverage_dir, - internal_getpid()); - CHECK_LE(res, tmp_path.size()); - fd_t map_fd = OpenFile(tmp_path.data(), WrOnly, &err); - if (map_fd == kInvalidFd) { - Report("Coverage: failed to open %s for writing: %d\n", tmp_path.data(), - err); - Die(); - } - - if (!WriteToFile(map_fd, text.data(), text.length(), nullptr, &err)) { - Printf("sancov.map write failed: %d\n", err); - Die(); - } - CloseFile(map_fd); - - InternalScopedString path(64 + internal_strlen(coverage_dir)); - res = internal_snprintf((char *)path.data(), path.size(), "%s/%zd.sancov.map", - coverage_dir, internal_getpid()); - CHECK_LE(res, path.size()); - if (!RenameFile(tmp_path.data(), path.data(), &err)) { - Printf("sancov.map rename failed: %d\n", err); - Die(); - } -} - -} // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dll_thunk.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dll_thunk.cc new file mode 100644 index 0000000..d5e459f --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dll_thunk.cc @@ -0,0 +1,21 @@ +//===-- sanitizer_coverage_win_dll_thunk.cc -------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines a family of thunks that should be statically linked into +// the DLLs that have instrumentation in order to delegate the calls to the +// shared runtime that lives in the main binary. +// See https://github.com/google/sanitizers/issues/209 for the details. +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DLL_THUNK +#include "sanitizer_win_dll_thunk.h" +// Sanitizer Coverage interface functions. +#define INTERFACE_FUNCTION(Name) INTERCEPT_SANITIZER_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "sanitizer_coverage_interface.inc" +#endif // SANITIZER_DLL_THUNK diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dynamic_runtime_thunk.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dynamic_runtime_thunk.cc new file mode 100644 index 0000000..988a206 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_dynamic_runtime_thunk.cc @@ -0,0 +1,21 @@ +//===-- sanitizer_coverage_win_dynamic_runtime_thunk.cc -------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines things that need to be present in the application modules +// to interact with Sanitizer Coverage, when it is included in a dll. +// +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC_RUNTIME_THUNK +#define SANITIZER_IMPORT_INTERFACE 1 +#include "sanitizer_win_defs.h" +// Define weak alias for all weak functions imported from sanitizer coverage. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) WIN_WEAK_IMPORT_DEF(Name) +#include "sanitizer_coverage_interface.inc" +#endif // SANITIZER_DYNAMIC_RUNTIME_THUNK diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_sections.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_sections.cc new file mode 100644 index 0000000..4b0bbf1 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_sections.cc @@ -0,0 +1,22 @@ +//===-- sanitizer_coverage_win_sections.cc --------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines delimiters for Sanitizer Coverage's section. +//===----------------------------------------------------------------------===// + +#include "sanitizer_platform.h" +#if SANITIZER_WINDOWS +#include <stdint.h> +#pragma section(".SCOV$A", read, write) // NOLINT +#pragma section(".SCOV$Z", read, write) // NOLINT +extern "C" { +__declspec(allocate(".SCOV$A")) uint32_t __start___sancov_guards = 0; +__declspec(allocate(".SCOV$Z")) uint32_t __stop___sancov_guards = 0; +} +#endif // SANITIZER_WINDOWS diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_weak_interception.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_weak_interception.cc new file mode 100644 index 0000000..0926f46 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_coverage_win_weak_interception.cc @@ -0,0 +1,24 @@ +//===-- sanitizer_coverage_win_weak_interception.cc -----------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This module should be included in Sanitizer Coverage when it implemented as a +// shared library on Windows (dll), in order to delegate the calls of weak +// functions to the implementation in the main executable when a strong +// definition is provided. +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC +#include "sanitizer_win_weak_interception.h" +#include "sanitizer_interface_internal.h" +#include "sancov_flags.h" +// Check if strong definitions for weak functions are present in the main +// executable. If that is the case, override dll functions to point to strong +// implementations. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "sanitizer_coverage_interface.inc" +#endif // SANITIZER_DYNAMIC diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.cc new file mode 100644 index 0000000..a6f9fc6 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.cc @@ -0,0 +1,35 @@ +//===-- sanitizer_errno.cc --------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is shared between sanitizers run-time libraries. +// +// Defines errno to avoid including errno.h and its dependencies into other +// files (e.g. interceptors are not supposed to include any system headers). +// +//===----------------------------------------------------------------------===// + +#include "sanitizer_errno_codes.h" +#include "sanitizer_internal_defs.h" + +#include <errno.h> + +namespace __sanitizer { + +COMPILER_CHECK(errno_ENOMEM == ENOMEM); +COMPILER_CHECK(errno_EBUSY == EBUSY); +COMPILER_CHECK(errno_EINVAL == EINVAL); + +// EOWNERDEAD is not present in some older platforms. +#if defined(EOWNERDEAD) +extern const int errno_EOWNERDEAD = EOWNERDEAD; +#else +extern const int errno_EOWNERDEAD = -1; +#endif + +} // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.h new file mode 100644 index 0000000..7872b89 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno.h @@ -0,0 +1,37 @@ +//===-- sanitizer_errno.h ---------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is shared between sanitizers run-time libraries. +// +// Defines errno to avoid including errno.h and its dependencies into sensitive +// files (e.g. interceptors are not supposed to include any system headers). +// It's ok to use errno.h directly when your file already depend on other system +// includes though. +// +//===----------------------------------------------------------------------===// + +#ifndef SANITIZER_ERRNO_H +#define SANITIZER_ERRNO_H + +#include "sanitizer_errno_codes.h" +#include "sanitizer_platform.h" + +#if SANITIZER_FREEBSD || SANITIZER_MAC +# define __errno_location __error +#elif SANITIZER_ANDROID +# define __errno_location __errno +#elif SANITIZER_WINDOWS +# define __errno_location _errno +#endif + +extern "C" int *__errno_location(); + +#define errno (*__errno_location()) + +#endif // SANITIZER_ERRNO_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno_codes.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno_codes.h new file mode 100644 index 0000000..dba774c --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_errno_codes.h @@ -0,0 +1,34 @@ +//===-- sanitizer_errno_codes.h ---------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is shared between sanitizers run-time libraries. +// +// Defines errno codes to avoid including errno.h and its dependencies into +// sensitive files (e.g. interceptors are not supposed to include any system +// headers). +// It's ok to use errno.h directly when your file already depend on other system +// includes though. +// +//===----------------------------------------------------------------------===// + +#ifndef SANITIZER_ERRNO_CODES_H +#define SANITIZER_ERRNO_CODES_H + +namespace __sanitizer { + +#define errno_ENOMEM 12 +#define errno_EBUSY 16 +#define errno_EINVAL 22 + +// Those might not present or their value differ on different platforms. +extern const int errno_EOWNERDEAD; + +} // namespace __sanitizer + +#endif // SANITIZER_ERRNO_CODES_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.h index 2477aed..4988fbb 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.h @@ -34,25 +34,46 @@ class FlagHandler : public FlagHandlerBase { bool Parse(const char *value) final; }; -template <> -inline bool FlagHandler<bool>::Parse(const char *value) { +inline bool ParseBool(const char *value, bool *b) { if (internal_strcmp(value, "0") == 0 || internal_strcmp(value, "no") == 0 || internal_strcmp(value, "false") == 0) { - *t_ = false; + *b = false; return true; } if (internal_strcmp(value, "1") == 0 || internal_strcmp(value, "yes") == 0 || internal_strcmp(value, "true") == 0) { - *t_ = true; + *b = true; return true; } + return false; +} + +template <> +inline bool FlagHandler<bool>::Parse(const char *value) { + if (ParseBool(value, t_)) return true; Printf("ERROR: Invalid value for bool option: '%s'\n", value); return false; } template <> +inline bool FlagHandler<HandleSignalMode>::Parse(const char *value) { + bool b; + if (ParseBool(value, &b)) { + *t_ = b ? kHandleSignalYes : kHandleSignalNo; + return true; + } + if (internal_strcmp(value, "2") == 0 || + internal_strcmp(value, "exclusive") == 0) { + *t_ = kHandleSignalExclusive; + return true; + } + Printf("ERROR: Invalid value for signal handler option: '%s'\n", value); + return false; +} + +template <> inline bool FlagHandler<const char *>::Parse(const char *value) { *t_ = internal_strdup(value); return true; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.h index 503126b..c2ec29d 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.h @@ -18,6 +18,12 @@ namespace __sanitizer { +enum HandleSignalMode { + kHandleSignalNo, + kHandleSignalYes, + kHandleSignalExclusive, +}; + struct CommonFlags { #define COMMON_FLAG(Type, Name, DefaultValue, Description) Type Name; #include "sanitizer_flags.inc" diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc index d7fa34a..9f71861 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc @@ -62,7 +62,7 @@ COMMON_FLAG( COMMON_FLAG( int, verbosity, 0, "Verbosity level (0 - silent, 1 - a bit of output, 2+ - more output).") -COMMON_FLAG(bool, detect_leaks, true, "Enable memory leak detection.") +COMMON_FLAG(bool, detect_leaks, !SANITIZER_MAC, "Enable memory leak detection.") COMMON_FLAG( bool, leak_check_at_exit, true, "Invoke leak checking in an atexit handler. Has no effect if " @@ -75,20 +75,27 @@ COMMON_FLAG(bool, print_summary, true, "If false, disable printing error summaries in addition to error " "reports.") COMMON_FLAG(int, print_module_map, 0, - "OS X only. 0 = don't print, 1 = print only once before process " - "exits, 2 = print after each report.") + "OS X only (0 - don't print, 1 - print only once before process " + "exits, 2 - print after each report).") COMMON_FLAG(bool, check_printf, true, "Check printf arguments.") -COMMON_FLAG(bool, handle_segv, true, - "If set, registers the tool's custom SIGSEGV/SIGBUS handler.") -COMMON_FLAG(bool, handle_abort, false, - "If set, registers the tool's custom SIGABRT handler.") -COMMON_FLAG(bool, handle_sigill, false, - "If set, registers the tool's custom SIGILL handler.") -COMMON_FLAG(bool, handle_sigfpe, true, - "If set, registers the tool's custom SIGFPE handler.") -COMMON_FLAG(bool, allow_user_segv_handler, false, - "If set, allows user to register a SEGV handler even if the tool " - "registers one.") +#define COMMON_FLAG_HANDLE_SIGNAL_HELP(signal) \ + "Controls custom tool's " #signal " handler (0 - do not registers the " \ + "handler, 1 - register the handler and allow user to set own, " \ + "2 - registers the handler and block user from changing it). " +COMMON_FLAG(HandleSignalMode, handle_segv, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGSEGV)) +COMMON_FLAG(HandleSignalMode, handle_sigbus, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGBUS)) +COMMON_FLAG(HandleSignalMode, handle_abort, kHandleSignalNo, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGABRT)) +COMMON_FLAG(HandleSignalMode, handle_sigill, kHandleSignalNo, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGILL)) +COMMON_FLAG(HandleSignalMode, handle_sigfpe, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGFPE)) +#undef COMMON_FLAG_HANDLE_SIGNAL_HELP +COMMON_FLAG(bool, allow_user_segv_handler, true, + "Deprecated. True has no effect, use handle_sigbus=1. If false, " + "handle_*=1 will be upgraded to handle_*=2.") COMMON_FLAG(bool, use_sigaltstack, true, "If set, uses alternate stack for signal handling.") COMMON_FLAG(bool, detect_deadlocks, false, @@ -134,22 +141,6 @@ COMMON_FLAG( bool, coverage, false, "If set, coverage information will be dumped at program shutdown (if the " "coverage instrumentation was enabled at compile time).") -COMMON_FLAG(bool, coverage_pcs, true, - "If set (and if 'coverage' is set too), the coverage information " - "will be dumped as a set of PC offsets for every module.") -COMMON_FLAG(bool, coverage_order_pcs, false, - "If true, the PCs will be dumped in the order they've" - " appeared during the execution.") -COMMON_FLAG(bool, coverage_bitset, false, - "If set (and if 'coverage' is set too), the coverage information " - "will also be dumped as a bitset to a separate file.") -COMMON_FLAG(bool, coverage_counters, false, - "If set (and if 'coverage' is set too), the bitmap that corresponds" - " to coverage counters will be dumped.") -COMMON_FLAG(bool, coverage_direct, SANITIZER_ANDROID, - "If set, coverage information will be dumped directly to a memory " - "mapped file. This way data is not lost even if the process is " - "suddenly killed.") COMMON_FLAG(const char *, coverage_dir, ".", "Target directory for coverage dumps. Defaults to the current " "directory.") @@ -190,12 +181,18 @@ COMMON_FLAG(bool, intercept_strstr, true, COMMON_FLAG(bool, intercept_strspn, true, "If set, uses custom wrappers for strspn and strcspn function " "to find more errors.") +COMMON_FLAG(bool, intercept_strtok, true, + "If set, uses a custom wrapper for the strtok function " + "to find more errors.") COMMON_FLAG(bool, intercept_strpbrk, true, "If set, uses custom wrappers for strpbrk function " "to find more errors.") COMMON_FLAG(bool, intercept_strlen, true, "If set, uses custom wrappers for strlen and strnlen functions " "to find more errors.") +COMMON_FLAG(bool, intercept_strndup, true, + "If set, uses custom wrappers for strndup functions " + "to find more errors.") COMMON_FLAG(bool, intercept_strchr, true, "If set, uses custom wrappers for strchr, strchrnul, and strrchr " "functions to find more errors.") diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_interface_internal.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_interface_internal.h index 174d5e9..b28d8f0 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_interface_internal.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_interface_internal.h @@ -69,6 +69,32 @@ extern "C" { int __sanitizer_get_module_and_offset_for_pc( __sanitizer::uptr pc, char *module_path, __sanitizer::uptr module_path_len, __sanitizer::uptr *pc_offset); - } // extern "C" + + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_cmp(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_cmp1(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_cmp2(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_cmp4(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_cmp8(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_switch(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_div4(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_div8(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_gep(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_pc_indir(); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_pc_guard(__sanitizer::u32*); + SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE + void __sanitizer_cov_trace_pc_guard_init(__sanitizer::u32*, + __sanitizer::u32*); +} // extern "C" #endif // SANITIZER_INTERFACE_INTERNAL_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h index 5338f79..f35b095 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h @@ -21,8 +21,11 @@ // Only use SANITIZER_*ATTRIBUTE* before the function return type! #if SANITIZER_WINDOWS +#if SANITIZER_IMPORT_INTERFACE +# define SANITIZER_INTERFACE_ATTRIBUTE __declspec(dllimport) +#else # define SANITIZER_INTERFACE_ATTRIBUTE __declspec(dllexport) -// FIXME find out what we need on Windows, if anything. +#endif # define SANITIZER_WEAK_ATTRIBUTE #elif SANITIZER_GO # define SANITIZER_INTERFACE_ATTRIBUTE @@ -32,11 +35,46 @@ # define SANITIZER_WEAK_ATTRIBUTE __attribute__((weak)) #endif -#if (SANITIZER_LINUX || SANITIZER_MAC || SANITIZER_WINDOWS) && !SANITIZER_GO +//--------------------------- WEAK FUNCTIONS ---------------------------------// +// When working with weak functions, to simplify the code and make it more +// portable, when possible define a default implementation using this macro: +// +// SANITIZER_INTERFACE_WEAK_DEF(<return_type>, <name>, <parameter list>) +// +// For example: +// SANITIZER_INTERFACE_WEAK_DEF(bool, compare, int a, int b) { return a > b; } +// +#if SANITIZER_WINDOWS +#include "sanitizer_win_defs.h" +# define SANITIZER_INTERFACE_WEAK_DEF(ReturnType, Name, ...) \ + WIN_WEAK_EXPORT_DEF(ReturnType, Name, __VA_ARGS__) +#else +# define SANITIZER_INTERFACE_WEAK_DEF(ReturnType, Name, ...) \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE \ + ReturnType Name(__VA_ARGS__) +#endif + +// SANITIZER_SUPPORTS_WEAK_HOOKS means that we support real weak functions that +// will evaluate to a null pointer when not defined. +#if (SANITIZER_LINUX || SANITIZER_MAC) && !SANITIZER_GO # define SANITIZER_SUPPORTS_WEAK_HOOKS 1 #else # define SANITIZER_SUPPORTS_WEAK_HOOKS 0 #endif +// For some weak hooks that will be called very often and we want to avoid the +// overhead of executing the default implementation when it is not necessary, +// we can use the flag SANITIZER_SUPPORTS_WEAK_HOOKS to only define the default +// implementation for platforms that doesn't support weak symbols. For example: +// +// #if !SANITIZER_SUPPORT_WEAK_HOOKS +// SANITIZER_INTERFACE_WEAK_DEF(bool, compare_hook, int a, int b) { +// return a > b; +// } +// #endif +// +// And then use it as: if (compare_hook) compare_hook(a, b); +//----------------------------------------------------------------------------// + // We can use .preinit_array section on Linux to call sanitizer initialization // functions very early in the process startup (unless PIC macro is defined). @@ -114,6 +152,12 @@ typedef u32 operator_new_size_type; # endif #endif +#if SANITIZER_MAC +// On Darwin, thread IDs are 64-bit even on 32-bit systems. +typedef u64 tid_t; +#else +typedef uptr tid_t; +#endif // ----------- ATTENTION ------------- // This header should NOT include any other headers to avoid portability issues. @@ -289,7 +333,12 @@ void NORETURN CheckFailed(const char *file, int line, const char *cond, enum LinkerInitialized { LINKER_INITIALIZED = 0 }; #if !defined(_MSC_VER) || defined(__clang__) +#if SANITIZER_S390_31 +#define GET_CALLER_PC() \ + (__sanitizer::uptr) __builtin_extract_return_addr(__builtin_return_address(0)) +#else #define GET_CALLER_PC() (__sanitizer::uptr) __builtin_return_address(0) +#endif #define GET_CURRENT_FRAME() (__sanitizer::uptr) __builtin_frame_address(0) inline void Trap() { __builtin_trap(); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc index 7328a5c..8c3c1e5 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc @@ -59,11 +59,17 @@ #include <ucontext.h> #include <unistd.h> +#if SANITIZER_LINUX +#include <sys/utsname.h> +#endif + +#if SANITIZER_LINUX && !SANITIZER_ANDROID +#include <sys/personality.h> +#endif + #if SANITIZER_FREEBSD #include <sys/exec.h> #include <sys/sysctl.h> -#include <vm/vm_param.h> -#include <vm/pmap.h> #include <machine/atomic.h> extern "C" { // <sys/umtx.h> must be included after <errno.h> and <sys/types.h> on @@ -77,6 +83,20 @@ extern char **environ; // provided by crt1 #include <sys/signal.h> #endif +#ifndef __GLIBC_PREREQ +#define __GLIBC_PREREQ(x, y) 0 +#endif + +#if SANITIZER_LINUX && __GLIBC_PREREQ(2, 16) +# define SANITIZER_USE_GETAUXVAL 1 +#else +# define SANITIZER_USE_GETAUXVAL 0 +#endif + +#if SANITIZER_USE_GETAUXVAL +#include <sys/auxv.h> +#endif + #if SANITIZER_LINUX // <linux/time.h> struct kernel_timeval { @@ -197,7 +217,6 @@ static void stat64_to_stat(struct stat64 *in, struct stat *out) { out->st_atime = in->st_atime; out->st_mtime = in->st_mtime; out->st_ctime = in->st_ctime; - out->st_ino = in->st_ino; } #endif @@ -217,13 +236,13 @@ static void kernel_stat_to_stat(struct kernel_stat *in, struct stat *out) { out->st_atime = in->st_atime_nsec; out->st_mtime = in->st_mtime_nsec; out->st_ctime = in->st_ctime_nsec; - out->st_ino = in->st_ino; } #endif uptr internal_stat(const char *path, void *buf) { #if SANITIZER_FREEBSD - return internal_syscall(SYSCALL(stat), path, buf); + return internal_syscall(SYSCALL(fstatat), AT_FDCWD, (uptr)path, + (uptr)buf, 0); #elif SANITIZER_USES_CANONICAL_LINUX_SYSCALLS return internal_syscall(SYSCALL(newfstatat), AT_FDCWD, (uptr)path, (uptr)buf, 0); @@ -247,7 +266,8 @@ uptr internal_stat(const char *path, void *buf) { uptr internal_lstat(const char *path, void *buf) { #if SANITIZER_FREEBSD - return internal_syscall(SYSCALL(lstat), path, buf); + return internal_syscall(SYSCALL(fstatat), AT_FDCWD, (uptr)path, + (uptr)buf, AT_SYMLINK_NOFOLLOW); #elif SANITIZER_USES_CANONICAL_LINUX_SYSCALLS return internal_syscall(SYSCALL(newfstatat), AT_FDCWD, (uptr)path, (uptr)buf, AT_SYMLINK_NOFOLLOW); @@ -370,7 +390,7 @@ bool FileExists(const char *filename) { return S_ISREG(st.st_mode); } -uptr GetTid() { +tid_t GetTid() { #if SANITIZER_FREEBSD return (uptr)pthread_self(); #else @@ -535,7 +555,7 @@ void BlockingMutex::Lock() { void BlockingMutex::Unlock() { atomic_uint32_t *m = reinterpret_cast<atomic_uint32_t *>(&opaque_storage_); - u32 v = atomic_exchange(m, MtxUnlocked, memory_order_relaxed); + u32 v = atomic_exchange(m, MtxUnlocked, memory_order_release); CHECK_NE(v, MtxUnlocked); if (v == MtxSleeping) { #if SANITIZER_FREEBSD @@ -590,7 +610,9 @@ uptr internal_getppid() { } uptr internal_getdents(fd_t fd, struct linux_dirent *dirp, unsigned int count) { -#if SANITIZER_USES_CANONICAL_LINUX_SYSCALLS +#if SANITIZER_FREEBSD + return internal_syscall(SYSCALL(getdirentries), fd, (uptr)dirp, count, NULL); +#elif SANITIZER_USES_CANONICAL_LINUX_SYSCALLS return internal_syscall(SYSCALL(getdents64), fd, (uptr)dirp, count); #else return internal_syscall(SYSCALL(getdents), fd, (uptr)dirp, count); @@ -607,8 +629,7 @@ uptr internal_prctl(int option, uptr arg2, uptr arg3, uptr arg4, uptr arg5) { } #endif -uptr internal_sigaltstack(const struct sigaltstack *ss, - struct sigaltstack *oss) { +uptr internal_sigaltstack(const void *ss, void *oss) { return internal_syscall(SYSCALL(sigaltstack), (uptr)ss, (uptr)oss); } @@ -799,12 +820,80 @@ bool ThreadLister::GetDirectoryEntries() { return true; } +#if SANITIZER_WORDSIZE == 32 +// Take care of unusable kernel area in top gigabyte. +static uptr GetKernelAreaSize() { +#if SANITIZER_LINUX && !SANITIZER_X32 + const uptr gbyte = 1UL << 30; + + // Firstly check if there are writable segments + // mapped to top gigabyte (e.g. stack). + MemoryMappingLayout proc_maps(/*cache_enabled*/true); + MemoryMappedSegment segment; + while (proc_maps.Next(&segment)) { + if ((segment.end >= 3 * gbyte) && segment.IsWritable()) return 0; + } + +#if !SANITIZER_ANDROID + // Even if nothing is mapped, top Gb may still be accessible + // if we are running on 64-bit kernel. + // Uname may report misleading results if personality type + // is modified (e.g. under schroot) so check this as well. + struct utsname uname_info; + int pers = personality(0xffffffffUL); + if (!(pers & PER_MASK) + && uname(&uname_info) == 0 + && internal_strstr(uname_info.machine, "64")) + return 0; +#endif // SANITIZER_ANDROID + + // Top gigabyte is reserved for kernel. + return gbyte; +#else + return 0; +#endif // SANITIZER_LINUX && !SANITIZER_X32 +} +#endif // SANITIZER_WORDSIZE == 32 + +uptr GetMaxVirtualAddress() { +#if SANITIZER_WORDSIZE == 64 +# if defined(__powerpc64__) || defined(__aarch64__) + // On PowerPC64 we have two different address space layouts: 44- and 46-bit. + // We somehow need to figure out which one we are using now and choose + // one of 0x00000fffffffffffUL and 0x00003fffffffffffUL. + // Note that with 'ulimit -s unlimited' the stack is moved away from the top + // of the address space, so simply checking the stack address is not enough. + // This should (does) work for both PowerPC64 Endian modes. + // Similarly, aarch64 has multiple address space layouts: 39, 42 and 47-bit. + return (1ULL << (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1)) - 1; +# elif defined(__mips64) + return (1ULL << 40) - 1; // 0x000000ffffffffffUL; +# elif defined(__s390x__) + return (1ULL << 53) - 1; // 0x001fffffffffffffUL; +# else + return (1ULL << 47) - 1; // 0x00007fffffffffffUL; +# endif +#else // SANITIZER_WORDSIZE == 32 +# if defined(__s390__) + return (1ULL << 31) - 1; // 0x7fffffff; +# else + uptr res = (1ULL << 32) - 1; // 0xffffffff; + if (!common_flags()->full_address_space) + res -= GetKernelAreaSize(); + CHECK_LT(reinterpret_cast<uptr>(&res), res); + return res; +# endif +#endif // SANITIZER_WORDSIZE +} + uptr GetPageSize() { // Android post-M sysconf(_SC_PAGESIZE) crashes if called from .preinit_array. #if SANITIZER_ANDROID return 4096; #elif SANITIZER_LINUX && (defined(__x86_64__) || defined(__i386__)) return EXEC_PAGESIZE; +#elif SANITIZER_USE_GETAUXVAL + return getauxval(AT_PAGESZ); #else return sysconf(_SC_PAGESIZE); // EXEC_PAGESIZE may not be trustworthy. #endif @@ -1097,36 +1186,50 @@ uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, int *parent_tidptr, void *newtls, int *child_tidptr) { long long res; -/* Stack frame offsets. */ -#if _CALL_ELF != 2 -#define FRAME_MIN_SIZE 112 -#define FRAME_TOC_SAVE 40 +// Stack frame structure. +#if SANITIZER_PPC64V1 +// Back chain == 0 (SP + 112) +// Frame (112 bytes): +// Parameter save area (SP + 48), 8 doublewords +// TOC save area (SP + 40) +// Link editor doubleword (SP + 32) +// Compiler doubleword (SP + 24) +// LR save area (SP + 16) +// CR save area (SP + 8) +// Back chain (SP + 0) +# define FRAME_SIZE 112 +# define FRAME_TOC_SAVE_OFFSET 40 +#elif SANITIZER_PPC64V2 +// Back chain == 0 (SP + 32) +// Frame (32 bytes): +// TOC save area (SP + 24) +// LR save area (SP + 16) +// CR save area (SP + 8) +// Back chain (SP + 0) +# define FRAME_SIZE 32 +# define FRAME_TOC_SAVE_OFFSET 24 #else -#define FRAME_MIN_SIZE 32 -#define FRAME_TOC_SAVE 24 +# error "Unsupported PPC64 ABI" #endif if (!fn || !child_stack) return -EINVAL; CHECK_EQ(0, (uptr)child_stack % 16); - child_stack = (char *)child_stack - 2 * sizeof(unsigned long long); - ((unsigned long long *)child_stack)[0] = (uptr)fn; - ((unsigned long long *)child_stack)[1] = (uptr)arg; register int (*__fn)(void *) __asm__("r3") = fn; register void *__cstack __asm__("r4") = child_stack; register int __flags __asm__("r5") = flags; - register void * __arg __asm__("r6") = arg; - register int * __ptidptr __asm__("r7") = parent_tidptr; - register void * __newtls __asm__("r8") = newtls; - register int * __ctidptr __asm__("r9") = child_tidptr; + register void *__arg __asm__("r6") = arg; + register int *__ptidptr __asm__("r7") = parent_tidptr; + register void *__newtls __asm__("r8") = newtls; + register int *__ctidptr __asm__("r9") = child_tidptr; __asm__ __volatile__( - /* fn, arg, child_stack are saved acrVoss the syscall */ + /* fn and arg are saved across the syscall */ "mr 28, %5\n\t" - "mr 29, %6\n\t" "mr 27, %8\n\t" /* syscall + r0 == __NR_clone r3 == flags r4 == child_stack r5 == parent_tidptr @@ -1144,15 +1247,21 @@ uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, "crandc cr1*4+eq, cr1*4+eq, cr0*4+so\n\t" "bne- cr1, 1f\n\t" + /* Set up stack frame */ + "li 29, 0\n\t" + "stdu 29, -8(1)\n\t" + "stdu 1, -%12(1)\n\t" /* Do the function call */ "std 2, %13(1)\n\t" -#if _CALL_ELF != 2 +#if SANITIZER_PPC64V1 "ld 0, 0(28)\n\t" "ld 2, 8(28)\n\t" "mtctr 0\n\t" -#else +#elif SANITIZER_PPC64V2 "mr 12, 28\n\t" "mtctr 12\n\t" +#else +# error "Unsupported PPC64 ABI" #endif "mr 3, 27\n\t" "bctrl\n\t" @@ -1166,13 +1275,151 @@ uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, "1:\n\t" "mr %0, 3\n\t" : "=r" (res) - : "0" (-1), "i" (EINVAL), - "i" (__NR_clone), "i" (__NR_exit), - "r" (__fn), "r" (__cstack), "r" (__flags), - "r" (__arg), "r" (__ptidptr), "r" (__newtls), - "r" (__ctidptr), "i" (FRAME_MIN_SIZE), "i" (FRAME_TOC_SAVE) - : "cr0", "cr1", "memory", "ctr", - "r0", "r29", "r27", "r28"); + : "0" (-1), + "i" (EINVAL), + "i" (__NR_clone), + "i" (__NR_exit), + "r" (__fn), + "r" (__cstack), + "r" (__flags), + "r" (__arg), + "r" (__ptidptr), + "r" (__newtls), + "r" (__ctidptr), + "i" (FRAME_SIZE), + "i" (FRAME_TOC_SAVE_OFFSET) + : "cr0", "cr1", "memory", "ctr", "r0", "r27", "r28", "r29"); + return res; +} +#elif defined(__i386__) && SANITIZER_LINUX +uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, + int *parent_tidptr, void *newtls, int *child_tidptr) { + int res; + if (!fn || !child_stack) + return -EINVAL; + CHECK_EQ(0, (uptr)child_stack % 16); + child_stack = (char *)child_stack - 7 * sizeof(unsigned int); + ((unsigned int *)child_stack)[0] = (uptr)flags; + ((unsigned int *)child_stack)[1] = (uptr)0; + ((unsigned int *)child_stack)[2] = (uptr)fn; + ((unsigned int *)child_stack)[3] = (uptr)arg; + __asm__ __volatile__( + /* %eax = syscall(%eax = SYSCALL(clone), + * %ebx = flags, + * %ecx = child_stack, + * %edx = parent_tidptr, + * %esi = new_tls, + * %edi = child_tidptr) + */ + + /* Obtain flags */ + "movl (%%ecx), %%ebx\n" + /* Do the system call */ + "pushl %%ebx\n" + "pushl %%esi\n" + "pushl %%edi\n" + /* Remember the flag value. */ + "movl %%ebx, (%%ecx)\n" + "int $0x80\n" + "popl %%edi\n" + "popl %%esi\n" + "popl %%ebx\n" + + /* if (%eax != 0) + * return; + */ + + "test %%eax,%%eax\n" + "jnz 1f\n" + + /* terminate the stack frame */ + "xorl %%ebp,%%ebp\n" + /* Call FN. */ + "call *%%ebx\n" +#ifdef PIC + "call here\n" + "here:\n" + "popl %%ebx\n" + "addl $_GLOBAL_OFFSET_TABLE_+[.-here], %%ebx\n" +#endif + /* Call exit */ + "movl %%eax, %%ebx\n" + "movl %2, %%eax\n" + "int $0x80\n" + "1:\n" + : "=a" (res) + : "a"(SYSCALL(clone)), "i"(SYSCALL(exit)), + "c"(child_stack), + "d"(parent_tidptr), + "S"(newtls), + "D"(child_tidptr) + : "memory"); + return res; +} +#elif defined(__arm__) && SANITIZER_LINUX +uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, + int *parent_tidptr, void *newtls, int *child_tidptr) { + unsigned int res; + if (!fn || !child_stack) + return -EINVAL; + child_stack = (char *)child_stack - 2 * sizeof(unsigned int); + ((unsigned int *)child_stack)[0] = (uptr)fn; + ((unsigned int *)child_stack)[1] = (uptr)arg; + register int r0 __asm__("r0") = flags; + register void *r1 __asm__("r1") = child_stack; + register int *r2 __asm__("r2") = parent_tidptr; + register void *r3 __asm__("r3") = newtls; + register int *r4 __asm__("r4") = child_tidptr; + register int r7 __asm__("r7") = __NR_clone; + +#if __ARM_ARCH > 4 || defined (__ARM_ARCH_4T__) +# define ARCH_HAS_BX +#endif +#if __ARM_ARCH > 4 +# define ARCH_HAS_BLX +#endif + +#ifdef ARCH_HAS_BX +# ifdef ARCH_HAS_BLX +# define BLX(R) "blx " #R "\n" +# else +# define BLX(R) "mov lr, pc; bx " #R "\n" +# endif +#else +# define BLX(R) "mov lr, pc; mov pc," #R "\n" +#endif + + __asm__ __volatile__( + /* %r0 = syscall(%r7 = SYSCALL(clone), + * %r0 = flags, + * %r1 = child_stack, + * %r2 = parent_tidptr, + * %r3 = new_tls, + * %r4 = child_tidptr) + */ + + /* Do the system call */ + "swi 0x0\n" + + /* if (%r0 != 0) + * return %r0; + */ + "cmp r0, #0\n" + "bne 1f\n" + + /* In the child, now. Call "fn(arg)". */ + "ldr r0, [sp, #4]\n" + "ldr ip, [sp], #8\n" + BLX(ip) + /* Call _exit(%r0). */ + "mov r7, %7\n" + "swi 0x0\n" + "1:\n" + "mov %0, r0\n" + : "=r"(res) + : "r"(r0), "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r7), + "i"(__NR_exit) + : "memory"); return res; } #endif // defined(__x86_64__) && SANITIZER_LINUX @@ -1220,14 +1467,27 @@ AndroidApiLevel AndroidGetApiLevel() { #endif -bool IsHandledDeadlySignal(int signum) { - if (common_flags()->handle_abort && signum == SIGABRT) - return true; - if (common_flags()->handle_sigill && signum == SIGILL) - return true; - if (common_flags()->handle_sigfpe && signum == SIGFPE) - return true; - return (signum == SIGSEGV || signum == SIGBUS) && common_flags()->handle_segv; +static HandleSignalMode GetHandleSignalModeImpl(int signum) { + switch (signum) { + case SIGABRT: + return common_flags()->handle_abort; + case SIGILL: + return common_flags()->handle_sigill; + case SIGFPE: + return common_flags()->handle_sigfpe; + case SIGSEGV: + return common_flags()->handle_segv; + case SIGBUS: + return common_flags()->handle_sigbus; + } + return kHandleSignalNo; +} + +HandleSignalMode GetHandleSignalMode(int signum) { + HandleSignalMode result = GetHandleSignalModeImpl(signum); + if (result == kHandleSignalYes && !common_flags()->allow_user_segv_handler) + return kHandleSignalExclusive; + return result; } #if !SANITIZER_GO @@ -1395,11 +1655,53 @@ void MaybeReexec() { void PrintModuleMap() { } -uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding) { +void CheckNoDeepBind(const char *filename, int flag) { +#ifdef RTLD_DEEPBIND + if (flag & RTLD_DEEPBIND) { + Report( + "You are trying to dlopen a %s shared library with RTLD_DEEPBIND flag" + " which is incompatibe with sanitizer runtime " + "(see https://github.com/google/sanitizers/issues/611 for details" + "). If you want to run %s library under sanitizers please remove " + "RTLD_DEEPBIND from dlopen flags.\n", + filename, filename); + Die(); + } +#endif +} + +uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding, + uptr *largest_gap_found) { UNREACHABLE("FindAvailableMemoryRange is not available"); return 0; } +bool GetRandom(void *buffer, uptr length) { + if (!buffer || !length || length > 256) + return false; +#if defined(__NR_getrandom) + static atomic_uint8_t skip_getrandom_syscall; + if (!atomic_load_relaxed(&skip_getrandom_syscall)) { + // Up to 256 bytes, getrandom will not be interrupted. + uptr res = internal_syscall(SYSCALL(getrandom), buffer, length, 0); + int rverrno = 0; + if (internal_iserror(res, &rverrno) && rverrno == ENOSYS) + atomic_store_relaxed(&skip_getrandom_syscall, 1); + else if (res == length) + return true; + } +#endif + uptr fd = internal_open("/dev/urandom", O_RDONLY); + if (internal_iserror(fd)) + return false; + // internal_read deals with EINTR. + uptr res = internal_read(fd, buffer, length); + if (internal_iserror(res)) + return false; + internal_close(fd); + return true; +} + } // namespace __sanitizer #endif // SANITIZER_FREEBSD || SANITIZER_LINUX diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.h index d4d0f47..11cad6b 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux.h @@ -21,7 +21,6 @@ #include "sanitizer_platform_limits_posix.h" struct link_map; // Opaque type returned by dlopen(). -struct sigaltstack; namespace __sanitizer { // Dirent structure for getdents(). Note that this structure is different from @@ -30,8 +29,7 @@ struct linux_dirent; // Syscall wrappers. uptr internal_getdents(fd_t fd, struct linux_dirent *dirp, unsigned int count); -uptr internal_sigaltstack(const struct sigaltstack* ss, - struct sigaltstack* oss); +uptr internal_sigaltstack(const void* ss, void* oss); uptr internal_sigprocmask(int how, __sanitizer_sigset_t *set, __sanitizer_sigset_t *oldset); @@ -48,7 +46,8 @@ int internal_sigaction_syscall(int signum, const void *act, void *oldact); #endif void internal_sigdelset(__sanitizer_sigset_t *set, int signum); #if defined(__x86_64__) || defined(__mips__) || defined(__aarch64__) \ - || defined(__powerpc64__) || defined(__s390__) + || defined(__powerpc64__) || defined(__s390__) || defined(__i386__) \ + || defined(__arm__) uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg, int *parent_tidptr, void *newtls, int *child_tidptr); #endif @@ -87,6 +86,46 @@ bool LibraryNameIs(const char *full_name, const char *base_name); // Call cb for each region mapped by map. void ForEachMappedRegion(link_map *map, void (*cb)(const void *, uptr)); + +#if SANITIZER_ANDROID + +#if defined(__aarch64__) +# define __get_tls() \ + ({ void** __v; __asm__("mrs %0, tpidr_el0" : "=r"(__v)); __v; }) +#elif defined(__arm__) +# define __get_tls() \ + ({ void** __v; __asm__("mrc p15, 0, %0, c13, c0, 3" : "=r"(__v)); __v; }) +#elif defined(__mips__) +// On mips32r1, this goes via a kernel illegal instruction trap that's +// optimized for v1. +# define __get_tls() \ + ({ register void** __v asm("v1"); \ + __asm__(".set push\n" \ + ".set mips32r2\n" \ + "rdhwr %0,$29\n" \ + ".set pop\n" : "=r"(__v)); \ + __v; }) +#elif defined(__i386__) +# define __get_tls() \ + ({ void** __v; __asm__("movl %%gs:0, %0" : "=r"(__v)); __v; }) +#elif defined(__x86_64__) +# define __get_tls() \ + ({ void** __v; __asm__("mov %%fs:0, %0" : "=r"(__v)); __v; }) +#else +#error "Unsupported architecture." +#endif + +// The Android Bionic team has allocated a TLS slot for TSan starting with N, +// given that Android currently doesn't support ELF TLS. It is used to store +// Sanitizers thread specific data. +static const int TLS_SLOT_TSAN = 8; + +ALWAYS_INLINE uptr *get_android_tls_ptr() { + return reinterpret_cast<uptr *>(&__get_tls()[TLS_SLOT_TSAN]); +} + +#endif // SANITIZER_ANDROID + } // namespace __sanitizer #endif // SANITIZER_FREEBSD || SANITIZER_LINUX diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc index f99f0b5..52196db 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc @@ -81,28 +81,25 @@ void GetThreadStackTopAndBottom(bool at_initialization, uptr *stack_top, // Find the mapping that contains a stack variable. MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr start, end, offset; + MemoryMappedSegment segment; uptr prev_end = 0; - while (proc_maps.Next(&start, &end, &offset, nullptr, 0, - /* protection */nullptr)) { - if ((uptr)&rl < end) - break; - prev_end = end; + while (proc_maps.Next(&segment)) { + if ((uptr)&rl < segment.end) break; + prev_end = segment.end; } - CHECK((uptr)&rl >= start && (uptr)&rl < end); + CHECK((uptr)&rl >= segment.start && (uptr)&rl < segment.end); // Get stacksize from rlimit, but clip it so that it does not overlap // with other mappings. uptr stacksize = rl.rlim_cur; - if (stacksize > end - prev_end) - stacksize = end - prev_end; + if (stacksize > segment.end - prev_end) stacksize = segment.end - prev_end; // When running with unlimited stack size, we still want to set some limit. // The unlimited stack size is caused by 'ulimit -s unlimited'. // Also, for some reason, GNU make spawns subprocesses with unlimited stack. if (stacksize > kMaxThreadStackSize) stacksize = kMaxThreadStackSize; - *stack_top = end; - *stack_bottom = end - stacksize; + *stack_top = segment.end; + *stack_bottom = segment.end - stacksize; return; } pthread_attr_t attr; @@ -113,7 +110,6 @@ void GetThreadStackTopAndBottom(bool at_initialization, uptr *stack_top, my_pthread_attr_getstack(&attr, &stackaddr, &stacksize); pthread_attr_destroy(&attr); - CHECK_LE(stacksize, kMaxThreadStackSize); // Sanity check. *stack_top = (uptr)stackaddr + stacksize; *stack_bottom = (uptr)stackaddr; } @@ -183,8 +179,8 @@ void InitTlsSize() { } #endif // !SANITIZER_FREEBSD && !SANITIZER_ANDROID && !SANITIZER_GO #if (defined(__x86_64__) || defined(__i386__) || defined(__mips__) \ - || defined(__aarch64__) || defined(__powerpc64__) || defined(__s390__)) \ - && SANITIZER_LINUX && !SANITIZER_ANDROID + || defined(__aarch64__) || defined(__powerpc64__) || defined(__s390__) \ + || defined(__arm__)) && SANITIZER_LINUX && !SANITIZER_ANDROID // sizeof(struct pthread) from glibc. static atomic_uintptr_t kThreadDescriptorSize; @@ -192,14 +188,14 @@ uptr ThreadDescriptorSize() { uptr val = atomic_load(&kThreadDescriptorSize, memory_order_relaxed); if (val) return val; -#if defined(__x86_64__) || defined(__i386__) +#if defined(__x86_64__) || defined(__i386__) || defined(__arm__) #ifdef _CS_GNU_LIBC_VERSION char buf[64]; uptr len = confstr(_CS_GNU_LIBC_VERSION, buf, sizeof(buf)); if (len < sizeof(buf) && internal_strncmp(buf, "glibc 2.", 8) == 0) { char *end; int minor = internal_simple_strtoll(buf + 8, &end, 10); - if (end != buf + 8 && (*end == '\0' || *end == '.')) { + if (end != buf + 8 && (*end == '\0' || *end == '.' || *end == '-')) { int patch = 0; if (*end == '.') // strtoll will return 0 if no valid conversion could be performed @@ -208,6 +204,9 @@ uptr ThreadDescriptorSize() { /* sizeof(struct pthread) values from various glibc versions. */ if (SANITIZER_X32) val = 1728; // Assume only one particular version for x32. + // For ARM sizeof(struct pthread) changed in Glibc 2.23. + else if (SANITIZER_ARM) + val = minor <= 22 ? 1120 : 1216; else if (minor <= 3) val = FIRST_32_SECOND_64(1104, 1696); else if (minor == 4) @@ -270,9 +269,7 @@ static uptr TlsPreTcbSize() { # endif const uptr kTlsAlign = 16; const uptr kTlsPreTcbSize = - (ThreadDescriptorSize() + kTcbHead + kTlsAlign - 1) & ~(kTlsAlign - 1); - InitTlsSize(); - g_tls_size = (g_tls_size + kTlsPreTcbSize + kTlsAlign -1) & ~(kTlsAlign - 1); + RoundUpTo(ThreadDescriptorSize() + kTcbHead, kTlsAlign); return kTlsPreTcbSize; } #endif @@ -295,7 +292,7 @@ uptr ThreadSelf() { rdhwr %0,$29;\ .set pop" : "=r" (thread_pointer)); descr_addr = thread_pointer - kTlsTcbOffset - TlsPreTcbSize(); -# elif defined(__aarch64__) +# elif defined(__aarch64__) || defined(__arm__) descr_addr = reinterpret_cast<uptr>(__builtin_thread_pointer()) - ThreadDescriptorSize(); # elif defined(__s390__) @@ -344,7 +341,8 @@ static void GetTls(uptr *addr, uptr *size) { *size = GetTlsSize(); *addr -= *size; *addr += ThreadDescriptorSize(); -# elif defined(__mips__) || defined(__aarch64__) || defined(__powerpc64__) +# elif defined(__mips__) || defined(__aarch64__) || defined(__powerpc64__) \ + || defined(__arm__) *addr = ThreadSelf(); *size = GetTlsSize(); # else @@ -379,6 +377,8 @@ uptr GetTlsSize() { uptr addr, size; GetTls(&addr, &size); return size; +#elif defined(__mips__) || defined(__powerpc64__) + return RoundUpTo(g_tls_size + TlsPreTcbSize(), 16); #else return g_tls_size; #endif @@ -443,7 +443,9 @@ static int dl_iterate_phdr_cb(dl_phdr_info *info, size_t size, void *arg) { uptr cur_beg = info->dlpi_addr + phdr->p_vaddr; uptr cur_end = cur_beg + phdr->p_memsz; bool executable = phdr->p_flags & PF_X; - cur_module.addAddressRange(cur_beg, cur_end, executable); + bool writable = phdr->p_flags & PF_W; + cur_module.addAddressRange(cur_beg, cur_end, executable, + writable); } } data->modules->push_back(cur_module); @@ -546,6 +548,13 @@ void LogMessageOnPrintf(const char *str) { WriteToSyslog(str); } +#if SANITIZER_ANDROID && __ANDROID_API__ >= 21 +extern "C" void android_set_abort_message(const char *msg); +void SetAbortMessage(const char *str) { android_set_abort_message(str); } +#else +void SetAbortMessage(const char *str) {} +#endif + #endif // SANITIZER_LINUX } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_s390.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_s390.cc index 053fd17..a6da82e 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_s390.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_linux_s390.cc @@ -136,6 +136,18 @@ static bool FixedCVE_2016_2143() { if (ptr[0] == '.') patch = internal_simple_strtoll(ptr+1, &ptr, 10); if (major < 3) { + if (major == 2 && minor == 6 && patch == 32 && ptr[0] == '-' && + internal_strstr(ptr, ".el6")) { + // Check RHEL6 + int r1 = internal_simple_strtoll(ptr+1, &ptr, 10); + if (r1 >= 657) // 2.6.32-657.el6 or later + return true; + if (r1 == 642 && ptr[0] == '.') { + int r2 = internal_simple_strtoll(ptr+1, &ptr, 10); + if (r2 >= 9) // 2.6.32-642.9.1.el6 or later + return true; + } + } // <3.0 is bad. return false; } else if (major == 3) { @@ -145,6 +157,18 @@ static bool FixedCVE_2016_2143() { // 3.12.58+ is OK. if (minor == 12 && patch >= 58) return true; + if (minor == 10 && patch == 0 && ptr[0] == '-' && + internal_strstr(ptr, ".el7")) { + // Check RHEL7 + int r1 = internal_simple_strtoll(ptr+1, &ptr, 10); + if (r1 >= 426) // 3.10.0-426.el7 or later + return true; + if (r1 == 327 && ptr[0] == '.') { + int r2 = internal_simple_strtoll(ptr+1, &ptr, 10); + if (r2 >= 27) // 3.10.0-327.27.1.el7 or later + return true; + } + } // Otherwise, bad. return false; } else if (major == 4) { @@ -154,6 +178,13 @@ static bool FixedCVE_2016_2143() { // 4.4.6+ is OK. if (minor == 4 && patch >= 6) return true; + if (minor == 4 && patch == 0 && ptr[0] == '-' && + internal_strstr(buf.version, "Ubuntu")) { + // Check Ubuntu 16.04 + int r1 = internal_simple_strtoll(ptr+1, &ptr, 10); + if (r1 >= 13) // 4.4.0-13 or later + return true; + } // Otherwise, OK if 4.5+. return minor >= 5; } else { diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_list.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_list.h index c78cb4c..598ce51 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_list.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_list.h @@ -70,6 +70,17 @@ struct IntrusiveList { size_--; } + void extract(Item *prev, Item *x) { + CHECK(!empty()); + CHECK_NE(prev, nullptr); + CHECK_NE(x, nullptr); + CHECK_EQ(prev->next, x); + prev->next = x->next; + if (last_ == x) + last_ = prev; + size_--; + } + Item *front() { return first_; } const Item *front() const { return first_; } Item *back() { return last_; } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.cc index b4f8ab5..1edd415 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.cc @@ -93,20 +93,22 @@ namespace __sanitizer { #include "sanitizer_syscall_generic.inc" -// Direct syscalls, don't call libmalloc hooks. +// Direct syscalls, don't call libmalloc hooks (but not available on 10.6). extern "C" void *__mmap(void *addr, size_t len, int prot, int flags, int fildes, - off_t off); -extern "C" int __munmap(void *, size_t); + off_t off) SANITIZER_WEAK_ATTRIBUTE; +extern "C" int __munmap(void *, size_t) SANITIZER_WEAK_ATTRIBUTE; // ---------------------- sanitizer_libc.h uptr internal_mmap(void *addr, size_t length, int prot, int flags, int fd, u64 offset) { if (fd == -1) fd = VM_MAKE_TAG(VM_MEMORY_ANALYSIS_TOOL); - return (uptr)__mmap(addr, length, prot, flags, fd, offset); + if (&__mmap) return (uptr)__mmap(addr, length, prot, flags, fd, offset); + return (uptr)mmap(addr, length, prot, flags, fd, offset); } uptr internal_munmap(void *addr, uptr length) { - return __munmap(addr, length); + if (&__munmap) return __munmap(addr, length); + return munmap(addr, length); } int internal_mprotect(void *addr, uptr length, int prot) { @@ -189,20 +191,23 @@ void internal_sigfillset(__sanitizer_sigset_t *set) { sigfillset(set); } uptr internal_sigprocmask(int how, __sanitizer_sigset_t *set, __sanitizer_sigset_t *oldset) { - return sigprocmask(how, set, oldset); + // Don't use sigprocmask here, because it affects all threads. + return pthread_sigmask(how, set, oldset); } -// Doesn't call pthread_atfork() handlers. -extern "C" pid_t __fork(void); +// Doesn't call pthread_atfork() handlers (but not available on 10.6). +extern "C" pid_t __fork(void) SANITIZER_WEAK_ATTRIBUTE; int internal_fork() { - return __fork(); + if (&__fork) + return __fork(); + return fork(); } int internal_forkpty(int *amaster) { int master, slave; if (openpty(&master, &slave, nullptr, nullptr, nullptr) == -1) return -1; - int pid = __fork(); + int pid = internal_fork(); if (pid == -1) { close(master); close(slave); @@ -248,9 +253,8 @@ bool FileExists(const char *filename) { return S_ISREG(st.st_mode); } -uptr GetTid() { - // FIXME: This can potentially get truncated on 32-bit, where uptr is 4 bytes. - uint64_t tid; +tid_t GetTid() { + tid_t tid; pthread_threadid_np(nullptr, &tid); return tid; } @@ -344,20 +348,16 @@ BlockingMutex::BlockingMutex() { void BlockingMutex::Lock() { CHECK(sizeof(OSSpinLock) <= sizeof(opaque_storage_)); CHECK_EQ(OS_SPINLOCK_INIT, 0); - CHECK_NE(owner_, (uptr)pthread_self()); + CHECK_EQ(owner_, 0); OSSpinLockLock((OSSpinLock*)&opaque_storage_); - CHECK(!owner_); - owner_ = (uptr)pthread_self(); } void BlockingMutex::Unlock() { - CHECK(owner_ == (uptr)pthread_self()); - owner_ = 0; OSSpinLockUnlock((OSSpinLock*)&opaque_storage_); } void BlockingMutex::CheckLocked() { - CHECK_EQ((uptr)pthread_self(), owner_); + CHECK_NE(*(OSSpinLock*)&opaque_storage_, 0); } u64 NanoTime() { @@ -371,6 +371,27 @@ uptr GetTlsSize() { void InitTlsSize() { } +uptr TlsBaseAddr() { + uptr segbase = 0; +#if defined(__x86_64__) + asm("movq %%gs:0,%0" : "=r"(segbase)); +#elif defined(__i386__) + asm("movl %%gs:0,%0" : "=r"(segbase)); +#endif + return segbase; +} + +// The size of the tls on darwin does not appear to be well documented, +// however the vm memory map suggests that it is 1024 uptrs in size, +// with a size of 0x2000 bytes on x86_64 and 0x1000 bytes on i386. +uptr TlsSize() { +#if defined(__x86_64__) || defined(__i386__) + return 1024 * sizeof(uptr); +#else + return 0; +#endif +} + void GetThreadStackAndTls(bool main, uptr *stk_addr, uptr *stk_size, uptr *tls_addr, uptr *tls_size) { #if !SANITIZER_GO @@ -378,8 +399,8 @@ void GetThreadStackAndTls(bool main, uptr *stk_addr, uptr *stk_size, GetThreadStackTopAndBottom(main, &stack_top, &stack_bottom); *stk_addr = stack_bottom; *stk_size = stack_top - stack_bottom; - *tls_addr = 0; - *tls_size = 0; + *tls_addr = TlsBaseAddr(); + *tls_size = TlsSize(); #else *stk_addr = 0; *stk_size = 0; @@ -394,15 +415,30 @@ void ListOfModules::init() { memory_mapping.DumpListOfModules(&modules_); } -bool IsHandledDeadlySignal(int signum) { +static HandleSignalMode GetHandleSignalModeImpl(int signum) { + switch (signum) { + case SIGABRT: + return common_flags()->handle_abort; + case SIGILL: + return common_flags()->handle_sigill; + case SIGFPE: + return common_flags()->handle_sigfpe; + case SIGSEGV: + return common_flags()->handle_segv; + case SIGBUS: + return common_flags()->handle_sigbus; + } + return kHandleSignalNo; +} + +HandleSignalMode GetHandleSignalMode(int signum) { + // Handling fatal signals on watchOS and tvOS devices is disallowed. if ((SANITIZER_WATCHOS || SANITIZER_TVOS) && !(SANITIZER_IOSSIM)) - // Handling fatal signals on watchOS and tvOS devices is disallowed. - return false; - if (common_flags()->handle_abort && signum == SIGABRT) - return true; - if (common_flags()->handle_sigill && signum == SIGILL) - return true; - return (signum == SIGSEGV || signum == SIGBUS) && common_flags()->handle_segv; + return kHandleSignalNo; + HandleSignalMode result = GetHandleSignalModeImpl(signum); + if (result == kHandleSignalYes && !common_flags()->allow_user_segv_handler) + return kHandleSignalExclusive; + return result; } MacosVersion cached_macos_version = MACOS_VERSION_UNINITIALIZED; @@ -764,9 +800,69 @@ char **GetArgv() { return *_NSGetArgv(); } +#if defined(__aarch64__) && SANITIZER_IOS && !SANITIZER_IOSSIM +// The task_vm_info struct is normally provided by the macOS SDK, but we need +// fields only available in 10.12+. Declare the struct manually to be able to +// build against older SDKs. +struct __sanitizer_task_vm_info { + mach_vm_size_t virtual_size; + integer_t region_count; + integer_t page_size; + mach_vm_size_t resident_size; + mach_vm_size_t resident_size_peak; + mach_vm_size_t device; + mach_vm_size_t device_peak; + mach_vm_size_t internal; + mach_vm_size_t internal_peak; + mach_vm_size_t external; + mach_vm_size_t external_peak; + mach_vm_size_t reusable; + mach_vm_size_t reusable_peak; + mach_vm_size_t purgeable_volatile_pmap; + mach_vm_size_t purgeable_volatile_resident; + mach_vm_size_t purgeable_volatile_virtual; + mach_vm_size_t compressed; + mach_vm_size_t compressed_peak; + mach_vm_size_t compressed_lifetime; + mach_vm_size_t phys_footprint; + mach_vm_address_t min_address; + mach_vm_address_t max_address; +}; +#define __SANITIZER_TASK_VM_INFO_COUNT ((mach_msg_type_number_t) \ + (sizeof(__sanitizer_task_vm_info) / sizeof(natural_t))) + +uptr GetTaskInfoMaxAddress() { + __sanitizer_task_vm_info vm_info = {}; + mach_msg_type_number_t count = __SANITIZER_TASK_VM_INFO_COUNT; + int err = task_info(mach_task_self(), TASK_VM_INFO, (int *)&vm_info, &count); + if (err == 0) { + return vm_info.max_address - 1; + } else { + // xnu cannot provide vm address limit + return 0x200000000 - 1; + } +} +#endif + +uptr GetMaxVirtualAddress() { +#if SANITIZER_WORDSIZE == 64 +# if defined(__aarch64__) && SANITIZER_IOS && !SANITIZER_IOSSIM + // Get the maximum VM address + static uptr max_vm = GetTaskInfoMaxAddress(); + CHECK(max_vm); + return max_vm; +# else + return (1ULL << 47) - 1; // 0x00007fffffffffffUL; +# endif +#else // SANITIZER_WORDSIZE == 32 + return (1ULL << 32) - 1; // 0xffffffff; +#endif // SANITIZER_WORDSIZE +} + uptr FindAvailableMemoryRange(uptr shadow_size, uptr alignment, - uptr left_padding) { + uptr left_padding, + uptr *largest_gap_found) { typedef vm_region_submap_short_info_data_64_t RegionInfo; enum { kRegionInfoSize = VM_REGION_SUBMAP_SHORT_INFO_COUNT_64 }; // Start searching for available memory region past PAGEZERO, which is @@ -777,6 +873,7 @@ uptr FindAvailableMemoryRange(uptr shadow_size, mach_vm_address_t address = start_address; mach_vm_address_t free_begin = start_address; kern_return_t kr = KERN_SUCCESS; + if (largest_gap_found) *largest_gap_found = 0; while (kr == KERN_SUCCESS) { mach_vm_size_t vmsize = 0; natural_t depth = 0; @@ -786,10 +883,15 @@ uptr FindAvailableMemoryRange(uptr shadow_size, (vm_region_info_t)&vminfo, &count); if (free_begin != address) { // We found a free region [free_begin..address-1]. - uptr shadow_address = RoundUpTo((uptr)free_begin + left_padding, - alignment); - if (shadow_address + shadow_size < (uptr)address) { - return shadow_address; + uptr gap_start = RoundUpTo((uptr)free_begin + left_padding, alignment); + uptr gap_end = RoundDownTo((uptr)address, alignment); + uptr gap_size = gap_end > gap_start ? gap_end - gap_start : 0; + if (shadow_size < gap_size) { + return gap_start; + } + + if (largest_gap_found && *largest_gap_found < gap_size) { + *largest_gap_found = gap_size; } } // Move to the next region. @@ -884,6 +986,15 @@ void PrintModuleMap() { Printf("End of module map.\n"); } +void CheckNoDeepBind(const char *filename, int flag) { + // Do nothing. +} + +// FIXME: implement on this platform. +bool GetRandom(void *buffer, uptr length) { + UNIMPLEMENTED(); +} + } // namespace __sanitizer #endif // SANITIZER_MAC diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.h index 636d9bf..3f1c68c 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac.h @@ -36,6 +36,8 @@ MacosVersion GetMacosVersion(); char **GetEnviron(); +void RestrictMemoryToMaxAddress(uptr max_address); + } // namespace __sanitizer extern "C" { diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac_libcdep.cc new file mode 100644 index 0000000..c95daa9 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mac_libcdep.cc @@ -0,0 +1,30 @@ +//===-- sanitizer_mac_libcdep.cc ------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is shared between various sanitizers' runtime libraries and +// implements OSX-specific functions. +//===----------------------------------------------------------------------===// + +#include "sanitizer_platform.h" +#if SANITIZER_MAC +#include "sanitizer_mac.h" + +#include <sys/mman.h> + +namespace __sanitizer { + +void RestrictMemoryToMaxAddress(uptr max_address) { + uptr size_to_mmap = GetMaxVirtualAddress() + 1 - max_address; + void *res = MmapFixedNoAccess(max_address, size_to_mmap, "high gap"); + CHECK(res != MAP_FAILED); +} + +} // namespace __sanitizer + +#endif // SANITIZER_MAC diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_malloc_mac.inc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_malloc_mac.inc index 6fbee07..5699c59 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_malloc_mac.inc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_malloc_mac.inc @@ -59,6 +59,9 @@ INTERCEPTOR(void, malloc_destroy_zone, malloc_zone_t *zone) { uptr allocated_size = RoundUpTo(sizeof(sanitizer_zone), page_size); mprotect(zone, allocated_size, PROT_READ | PROT_WRITE); } + if (zone->zone_name) { + COMMON_MALLOC_FREE((void *)zone->zone_name); + } COMMON_MALLOC_FREE(zone); } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mutex.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mutex.h index d06fc45..1759bf1 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mutex.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_mutex.h @@ -83,6 +83,14 @@ class BlockingMutex { BlockingMutex(); void Lock(); void Unlock(); + + // This function does not guarantee an explicit check that the calling thread + // is the thread which owns the mutex. This behavior, while more strictly + // correct, causes problems in cases like StopTheWorld, where a parent thread + // owns the mutex but a child checks that it is locked. Rather than + // maintaining complex state to work around those situations, the check only + // checks that the mutex is owned, and assumes callers to be generally + // well-behaved. void CheckLocked(); private: uptr opaque_storage_[10]; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform.h index d9a8e8d..396f7c9 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform.h @@ -13,7 +13,7 @@ #ifndef SANITIZER_PLATFORM_H #define SANITIZER_PLATFORM_H -#if !defined(__linux__) && !defined(__FreeBSD__) && \ +#if !defined(__linux__) && !defined(__FreeBSD__) && !defined(__NetBSD__) && \ !defined(__APPLE__) && !defined(_WIN32) # error "This operating system is not supported" #endif @@ -30,6 +30,12 @@ # define SANITIZER_FREEBSD 0 #endif +#if defined(__NetBSD__) +# define SANITIZER_NETBSD 1 +#else +# define SANITIZER_NETBSD 0 +#endif + #if defined(__APPLE__) # define SANITIZER_MAC 1 # include <TargetConditionals.h> @@ -79,7 +85,8 @@ # define SANITIZER_ANDROID 0 #endif -#define SANITIZER_POSIX (SANITIZER_FREEBSD || SANITIZER_LINUX || SANITIZER_MAC) +#define SANITIZER_POSIX \ + (SANITIZER_FREEBSD || SANITIZER_LINUX || SANITIZER_MAC || SANITIZER_NETBSD) #if __LP64__ || defined(_WIN64) # define SANITIZER_WORDSIZE 64 @@ -162,6 +169,12 @@ # define SANITIZER_PPC64V2 0 #endif +#if defined(__arm__) +# define SANITIZER_ARM 1 +#else +# define SANITIZER_ARM 0 +#endif + // By default we allow to use SizeClassAllocator64 on 64-bit platform. // But in some cases (e.g. AArch64's 39-bit address space) SizeClassAllocator64 // does not work well and we need to fallback to SizeClassAllocator32. @@ -253,4 +266,15 @@ # define SANITIZER_GO 0 #endif +// On PowerPC and ARM Thumb, calling pthread_exit() causes LSan to detect leaks. +// pthread_exit() performs unwinding that leads to dlopen'ing libgcc_s.so. +// dlopen mallocs "libgcc_s.so" string which confuses LSan, it fails to realize +// that this allocation happens in dynamic linker and should be ignored. +#if SANITIZER_PPC || defined(__thumb__) +# define SANITIZER_SUPPRESS_LEAK_ON_PTHREAD_EXIT 1 +#else +# define SANITIZER_SUPPRESS_LEAK_ON_PTHREAD_EXIT 0 +#endif + + #endif // SANITIZER_PLATFORM_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h index 62875d1..0380cee 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h @@ -17,12 +17,20 @@ #include "sanitizer_internal_defs.h" #if !SANITIZER_WINDOWS +# define SI_WINDOWS 0 # define SI_NOT_WINDOWS 1 # include "sanitizer_platform_limits_posix.h" #else +# define SI_WINDOWS 1 # define SI_NOT_WINDOWS 0 #endif +#if SANITIZER_POSIX +# define SI_POSIX 1 +#else +# define SI_POSIX 0 +#endif + #if SANITIZER_LINUX && !SANITIZER_ANDROID # define SI_LINUX_NOT_ANDROID 1 #else @@ -41,6 +49,12 @@ # define SI_FREEBSD 0 #endif +#if SANITIZER_NETBSD +# define SI_NETBSD 1 +#else +# define SI_NETBSD 0 +#endif + #if SANITIZER_LINUX # define SI_LINUX 1 #else @@ -67,11 +81,18 @@ # define SI_UNIX_NOT_MAC 0 #endif +#if SANITIZER_LINUX && !SANITIZER_FREEBSD +# define SI_LINUX_NOT_FREEBSD 1 +# else +# define SI_LINUX_NOT_FREEBSD 0 +#endif + #define SANITIZER_INTERCEPT_STRLEN 1 #define SANITIZER_INTERCEPT_STRNLEN SI_NOT_MAC #define SANITIZER_INTERCEPT_STRCMP 1 #define SANITIZER_INTERCEPT_STRSTR 1 #define SANITIZER_INTERCEPT_STRCASESTR SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_STRTOK 1 #define SANITIZER_INTERCEPT_STRCHR 1 #define SANITIZER_INTERCEPT_STRCHRNUL SI_UNIX_NOT_MAC #define SANITIZER_INTERCEPT_STRRCHR 1 @@ -83,23 +104,37 @@ #define SANITIZER_INTERCEPT_MEMMOVE 1 #define SANITIZER_INTERCEPT_MEMCPY 1 #define SANITIZER_INTERCEPT_MEMCMP 1 +#define SANITIZER_INTERCEPT_STRNDUP SI_POSIX +#define SANITIZER_INTERCEPT___STRNDUP SI_LINUX_NOT_FREEBSD +#if defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__) && \ + __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ < 1070 +# define SI_MAC_DEPLOYMENT_BELOW_10_7 1 +#else +# define SI_MAC_DEPLOYMENT_BELOW_10_7 0 +#endif +// memmem on Darwin doesn't exist on 10.6 // FIXME: enable memmem on Windows. -#define SANITIZER_INTERCEPT_MEMMEM SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_MEMMEM \ + (SI_NOT_WINDOWS && !SI_MAC_DEPLOYMENT_BELOW_10_7) #define SANITIZER_INTERCEPT_MEMCHR 1 -#define SANITIZER_INTERCEPT_MEMRCHR SI_FREEBSD || SI_LINUX +#define SANITIZER_INTERCEPT_MEMRCHR (SI_FREEBSD || SI_LINUX || SI_NETBSD) #define SANITIZER_INTERCEPT_READ SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PREAD SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_WRITE SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PWRITE SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_FREAD SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_FWRITE SI_NOT_WINDOWS + #define SANITIZER_INTERCEPT_PREAD64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PWRITE64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_READV SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_WRITEV SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_PREADV SI_FREEBSD || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_PREADV \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_PWRITEV SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PREADV64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PWRITEV64 SI_LINUX_NOT_ANDROID @@ -114,7 +149,7 @@ #ifndef SANITIZER_INTERCEPT_PRINTF # define SANITIZER_INTERCEPT_PRINTF SI_NOT_WINDOWS -# define SANITIZER_INTERCEPT_PRINTF_L SI_FREEBSD +# define SANITIZER_INTERCEPT_PRINTF_L (SI_FREEBSD || SI_NETBSD) # define SANITIZER_INTERCEPT_ISOC99_PRINTF SI_LINUX_NOT_ANDROID #endif @@ -123,13 +158,14 @@ #define SANITIZER_INTERCEPT_GETPWNAM_AND_FRIENDS SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_GETPWNAM_R_AND_FRIENDS \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_GETPWENT \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_FGETPWENT SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_GETPWENT_R SI_FREEBSD || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_SETPWENT SI_MAC || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_CLOCK_GETTIME SI_FREEBSD || SI_LINUX +#define SANITIZER_INTERCEPT_GETPWENT_R \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_SETPWENT (SI_MAC || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_CLOCK_GETTIME (SI_FREEBSD || SI_NETBSD || SI_LINUX) #define SANITIZER_INTERCEPT_GETITIMER SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_TIME SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_GLOB SI_LINUX_NOT_ANDROID @@ -140,10 +176,11 @@ #define SANITIZER_INTERCEPT_GETNAMEINFO SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_GETSOCKNAME SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_GETHOSTBYNAME SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_GETHOSTBYNAME_R SI_FREEBSD || SI_LINUX -#define SANITIZER_INTERCEPT_GETHOSTBYNAME2_R SI_FREEBSD || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_GETHOSTBYADDR_R SI_FREEBSD || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_GETHOSTENT_R SI_FREEBSD || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_GETHOSTBYNAME_R (SI_FREEBSD || SI_LINUX) +#define SANITIZER_INTERCEPT_GETHOSTBYNAME2_R \ + (SI_FREEBSD || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_GETHOSTBYADDR_R (SI_FREEBSD || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_GETHOSTENT_R (SI_FREEBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_GETSOCKOPT SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_ACCEPT SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_ACCEPT4 SI_LINUX_NOT_ANDROID @@ -169,63 +206,67 @@ #define SANITIZER_INTERCEPT_GET_CURRENT_DIR_NAME SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_STRTOIMAX SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_MBSTOWCS SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_MBSNRTOWCS SI_MAC || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_MBSNRTOWCS (SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_WCSTOMBS SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_WCSNRTOMBS \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_WCRTOMB \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_TCGETATTR SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_REALPATH SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_CANONICALIZE_FILE_NAME SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_CONFSTR \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_SCHED_GETAFFINITY SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_SCHED_GETPARAM SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_STRERROR SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_STRERROR_R SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_XPG_STRERROR_R SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_SCANDIR \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_SCANDIR64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_GETGROUPS SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_POLL SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PPOLL SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_WORDEXP \ - SI_FREEBSD || (SI_MAC && !SI_IOS) || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || (SI_MAC && !SI_IOS) || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_SIGWAIT SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_SIGWAITINFO SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_SIGTIMEDWAIT SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_SIGSETOPS \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_SIGPENDING SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_SIGPROCMASK SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_BACKTRACE SI_FREEBSD || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_BACKTRACE \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_GETMNTENT SI_LINUX #define SANITIZER_INTERCEPT_GETMNTENT_R SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_STATFS SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_STATFS \ + (SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_STATFS64 \ - (SI_MAC && !SI_IOS) || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_STATVFS SI_FREEBSD || SI_LINUX_NOT_ANDROID + ((SI_MAC && !SI_IOS) || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_STATVFS \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_STATVFS64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_INITGROUPS SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_ETHER_NTOA_ATON SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_ETHER_HOST \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_ETHER_R SI_FREEBSD || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_SHMCTL \ - ((SI_FREEBSD || SI_LINUX_NOT_ANDROID) && SANITIZER_WORDSIZE == 64) + (SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_ETHER_R (SI_FREEBSD || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_SHMCTL \ + (SI_NETBSD || ((SI_FREEBSD || SI_LINUX_NOT_ANDROID) && \ + SANITIZER_WORDSIZE == 64)) // NOLINT #define SANITIZER_INTERCEPT_RANDOM_R SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PTHREAD_ATTR_GET SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PTHREAD_ATTR_GETINHERITSCHED \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_PTHREAD_ATTR_GETAFFINITY_NP SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETPSHARED SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETTYPE SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETPROTOCOL \ - SI_MAC || SI_LINUX_NOT_ANDROID + (SI_MAC || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETPRIOCEILING \ - SI_MAC || SI_LINUX_NOT_ANDROID + (SI_MAC || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETROBUST SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GETROBUST_NP SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_PTHREAD_RWLOCKATTR_GETPSHARED SI_NOT_WINDOWS @@ -240,33 +281,36 @@ #define SANITIZER_INTERCEPT_SINCOS SI_LINUX #define SANITIZER_INTERCEPT_REMQUO SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_LGAMMA SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_LGAMMA_R SI_FREEBSD || SI_LINUX +#define SANITIZER_INTERCEPT_LGAMMA_R (SI_FREEBSD || SI_LINUX) #define SANITIZER_INTERCEPT_LGAMMAL_R SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_DRAND48_R SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_RAND_R \ - SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_ICONV SI_FREEBSD || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_MAC || SI_LINUX_NOT_ANDROID) +#define SANITIZER_INTERCEPT_ICONV \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_TIMES SI_NOT_WINDOWS // FIXME: getline seems to be available on OSX 10.7 -#define SANITIZER_INTERCEPT_GETLINE SI_FREEBSD || SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_GETLINE \ + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) -#define SANITIZER_INTERCEPT__EXIT SI_LINUX || SI_FREEBSD || SI_MAC +#define SANITIZER_INTERCEPT__EXIT \ + (SI_LINUX || SI_FREEBSD || SI_NETBSD || SI_MAC) #define SANITIZER_INTERCEPT_PHTREAD_MUTEX SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_PTHREAD_SETNAME_NP \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_TLS_GET_ADDR \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID) #define SANITIZER_INTERCEPT_LISTXATTR SI_LINUX #define SANITIZER_INTERCEPT_GETXATTR SI_LINUX #define SANITIZER_INTERCEPT_GETRESID SI_LINUX #define SANITIZER_INTERCEPT_GETIFADDRS \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID || SI_MAC + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID || SI_MAC) #define SANITIZER_INTERCEPT_IF_INDEXTONAME \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID || SI_MAC + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID || SI_MAC) #define SANITIZER_INTERCEPT_CAPGET SI_LINUX_NOT_ANDROID #if SI_LINUX && defined(__arm__) #define SANITIZER_INTERCEPT_AEABI_MEM 1 @@ -274,51 +318,64 @@ #define SANITIZER_INTERCEPT_AEABI_MEM 0 #endif #define SANITIZER_INTERCEPT___BZERO SI_MAC -#define SANITIZER_INTERCEPT_FTIME !SI_FREEBSD && SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_FTIME (!SI_FREEBSD && !SI_NETBSD && SI_NOT_WINDOWS) #define SANITIZER_INTERCEPT_XDR SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_TSEARCH SI_LINUX_NOT_ANDROID || SI_MAC +#define SANITIZER_INTERCEPT_TSEARCH \ + (SI_LINUX_NOT_ANDROID || SI_MAC || SI_NETBSD) #define SANITIZER_INTERCEPT_LIBIO_INTERNALS SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_FOPEN SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_FOPEN64 SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_OPEN_MEMSTREAM SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_OPEN_MEMSTREAM (SI_LINUX_NOT_ANDROID || SI_NETBSD) #define SANITIZER_INTERCEPT_OBSTACK SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_FFLUSH SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_FCLOSE SI_NOT_WINDOWS #ifndef SANITIZER_INTERCEPT_DLOPEN_DLCLOSE #define SANITIZER_INTERCEPT_DLOPEN_DLCLOSE \ - SI_FREEBSD || SI_LINUX_NOT_ANDROID || SI_MAC + (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID || SI_MAC) #endif -#define SANITIZER_INTERCEPT_GETPASS SI_LINUX_NOT_ANDROID || SI_MAC +#define SANITIZER_INTERCEPT_GETPASS \ + (SI_LINUX_NOT_ANDROID || SI_MAC || SI_NETBSD) #define SANITIZER_INTERCEPT_TIMERFD SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT_MLOCKX SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_FOPENCOOKIE SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_SEM SI_LINUX || SI_FREEBSD +#define SANITIZER_INTERCEPT_SEM (SI_LINUX || SI_FREEBSD || SI_NETBSD) #define SANITIZER_INTERCEPT_PTHREAD_SETCANCEL SI_NOT_WINDOWS -#define SANITIZER_INTERCEPT_MINCORE SI_LINUX +#define SANITIZER_INTERCEPT_MINCORE (SI_LINUX || SI_NETBSD) #define SANITIZER_INTERCEPT_PROCESS_VM_READV SI_LINUX -#define SANITIZER_INTERCEPT_CTERMID SI_LINUX || SI_MAC || SI_FREEBSD -#define SANITIZER_INTERCEPT_CTERMID_R SI_MAC || SI_FREEBSD +#define SANITIZER_INTERCEPT_CTERMID \ + (SI_LINUX || SI_MAC || SI_FREEBSD || SI_NETBSD) +#define SANITIZER_INTERCEPT_CTERMID_R (SI_MAC || SI_FREEBSD) -#define SANITIZER_INTERCEPTOR_HOOKS SI_LINUX +#define SANITIZER_INTERCEPTOR_HOOKS (SI_LINUX || SI_MAC || SI_WINDOWS) #define SANITIZER_INTERCEPT_RECV_RECVFROM SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_SEND_SENDTO SI_NOT_WINDOWS #define SANITIZER_INTERCEPT_EVENTFD_READ_WRITE SI_LINUX -#define SANITIZER_INTERCEPT_STAT (SI_FREEBSD || SI_MAC || SI_ANDROID) -#define SANITIZER_INTERCEPT___XSTAT !SANITIZER_INTERCEPT_STAT && SI_NOT_WINDOWS +#define SANITIZER_INTERCEPT_STAT \ + (SI_FREEBSD || SI_MAC || SI_ANDROID || SI_NETBSD) +#define SANITIZER_INTERCEPT___XSTAT \ + (!SANITIZER_INTERCEPT_STAT && SI_NOT_WINDOWS) #define SANITIZER_INTERCEPT___XSTAT64 SI_LINUX_NOT_ANDROID #define SANITIZER_INTERCEPT___LXSTAT SANITIZER_INTERCEPT___XSTAT #define SANITIZER_INTERCEPT___LXSTAT64 SI_LINUX_NOT_ANDROID -#define SANITIZER_INTERCEPT_UTMP SI_NOT_WINDOWS && !SI_MAC && !SI_FREEBSD -#define SANITIZER_INTERCEPT_UTMPX SI_LINUX_NOT_ANDROID || SI_MAC || SI_FREEBSD - -#define SANITIZER_INTERCEPT_MALLOPT_AND_MALLINFO (!SI_FREEBSD && !SI_MAC) -#define SANITIZER_INTERCEPT_MEMALIGN (!SI_FREEBSD && !SI_MAC) -#define SANITIZER_INTERCEPT_PVALLOC (!SI_FREEBSD && !SI_MAC) -#define SANITIZER_INTERCEPT_CFREE (!SI_FREEBSD && !SI_MAC) +#define SANITIZER_INTERCEPT_UTMP (SI_NOT_WINDOWS && !SI_MAC && !SI_FREEBSD) +#define SANITIZER_INTERCEPT_UTMPX (SI_LINUX_NOT_ANDROID || SI_MAC || SI_FREEBSD) + +#define SANITIZER_INTERCEPT_GETLOADAVG \ + (SI_LINUX_NOT_ANDROID || SI_MAC || SI_FREEBSD || SI_NETBSD) + +#define SANITIZER_INTERCEPT_MALLOPT_AND_MALLINFO \ + (!SI_FREEBSD && !SI_MAC && !SI_NETBSD) +#define SANITIZER_INTERCEPT_MEMALIGN (!SI_FREEBSD && !SI_MAC && !SI_NETBSD) +#define SANITIZER_INTERCEPT_PVALLOC (!SI_FREEBSD && !SI_MAC && !SI_NETBSD) +#define SANITIZER_INTERCEPT_CFREE (!SI_FREEBSD && !SI_MAC && !SI_NETBSD) +#define SANITIZER_INTERCEPT_ALIGNED_ALLOC (!SI_MAC) +#define SANITIZER_INTERCEPT_MALLOC_USABLE_SIZE (!SI_MAC) +#define SANITIZER_INTERCEPT_MCHECK_MPROBE SI_LINUX_NOT_ANDROID +#define SANITIZER_INTERCEPT_WCSCAT SI_NOT_WINDOWS #endif // #ifndef SANITIZER_PLATFORM_INTERCEPTORS_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cc index 683f019..83f4fd2 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cc @@ -25,7 +25,6 @@ #endif #include <arpa/inet.h> #include <dirent.h> -#include <errno.h> #include <grp.h> #include <limits.h> #include <net/if.h> @@ -931,14 +930,6 @@ unsigned struct_ElfW_Phdr_sz = sizeof(Elf_Phdr); unsigned IOCTL_SNDCTL_DSP_GETOSPACE = SNDCTL_DSP_GETOSPACE; #endif // (SANITIZER_LINUX || SANITIZER_FREEBSD) && !SANITIZER_ANDROID - const int errno_EINVAL = EINVAL; -// EOWNERDEAD is not present in some older platforms. -#if defined(EOWNERDEAD) - const int errno_EOWNERDEAD = EOWNERDEAD; -#else - const int errno_EOWNERDEAD = -1; -#endif - const int si_SEGV_MAPERR = SEGV_MAPERR; const int si_SEGV_ACCERR = SEGV_ACCERR; } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h index 5cbd78d..02c5ef6 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h @@ -23,6 +23,9 @@ // incorporates the map structure. # define GET_LINK_MAP_BY_DLOPEN_HANDLE(handle) \ ((link_map*)((handle) == nullptr ? nullptr : ((char*)(handle) + 544))) +// Get sys/_types.h, because that tells us whether 64-bit inodes are +// used in struct dirent below. +#include <sys/_types.h> #else # define GET_LINK_MAP_BY_DLOPEN_HANDLE(handle) ((link_map*)(handle)) #endif // !SANITIZER_FREEBSD @@ -87,7 +90,7 @@ namespace __sanitizer { #elif defined(__mips__) const unsigned struct_kernel_stat_sz = SANITIZER_ANDROID ? FIRST_32_SECOND_64(104, 128) : - FIRST_32_SECOND_64(144, 216); + FIRST_32_SECOND_64(160, 216); const unsigned struct_kernel_stat64_sz = 104; #elif defined(__s390__) && !defined(__s390x__) const unsigned struct_kernel_stat_sz = 64; @@ -489,7 +492,12 @@ namespace __sanitizer { }; #elif SANITIZER_FREEBSD struct __sanitizer_dirent { +#if defined(__INO64) + unsigned long long d_fileno; + unsigned long long d_off; +#else unsigned int d_fileno; +#endif unsigned short d_reclen; // more fields that we don't care about }; @@ -1460,9 +1468,6 @@ struct __sanitizer_cookie_io_functions_t { extern unsigned IOCTL_PIO_SCRNMAP; #endif - extern const int errno_EINVAL; - extern const int errno_EOWNERDEAD; - extern const int si_SEGV_MAPERR; extern const int si_SEGV_ACCERR; } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc index c70d5a4..8d3128a 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc @@ -22,18 +22,11 @@ #include "sanitizer_procmaps.h" #include "sanitizer_stacktrace.h" +#include <errno.h> #include <fcntl.h> #include <signal.h> #include <sys/mman.h> -#if SANITIZER_LINUX -#include <sys/utsname.h> -#endif - -#if SANITIZER_LINUX && !SANITIZER_ANDROID -#include <sys/personality.h> -#endif - #if SANITIZER_FREEBSD // The MAP_NORESERVE define has been removed in FreeBSD 11.x, and even before // that, it was never implemented. So just define it to zero. @@ -48,87 +41,13 @@ uptr GetMmapGranularity() { return GetPageSize(); } -#if SANITIZER_WORDSIZE == 32 -// Take care of unusable kernel area in top gigabyte. -static uptr GetKernelAreaSize() { -#if SANITIZER_LINUX && !SANITIZER_X32 - const uptr gbyte = 1UL << 30; - - // Firstly check if there are writable segments - // mapped to top gigabyte (e.g. stack). - MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr end, prot; - while (proc_maps.Next(/*start*/nullptr, &end, - /*offset*/nullptr, /*filename*/nullptr, - /*filename_size*/0, &prot)) { - if ((end >= 3 * gbyte) - && (prot & MemoryMappingLayout::kProtectionWrite) != 0) - return 0; - } - -#if !SANITIZER_ANDROID - // Even if nothing is mapped, top Gb may still be accessible - // if we are running on 64-bit kernel. - // Uname may report misleading results if personality type - // is modified (e.g. under schroot) so check this as well. - struct utsname uname_info; - int pers = personality(0xffffffffUL); - if (!(pers & PER_MASK) - && uname(&uname_info) == 0 - && internal_strstr(uname_info.machine, "64")) - return 0; -#endif // SANITIZER_ANDROID - - // Top gigabyte is reserved for kernel. - return gbyte; -#else - return 0; -#endif // SANITIZER_LINUX && !SANITIZER_X32 -} -#endif // SANITIZER_WORDSIZE == 32 - -uptr GetMaxVirtualAddress() { -#if SANITIZER_WORDSIZE == 64 -# if defined(__aarch64__) && SANITIZER_IOS && !SANITIZER_IOSSIM - // Ideally, we would derive the upper bound from MACH_VM_MAX_ADDRESS. The - // upper bound can change depending on the device. - return 0x200000000 - 1; -# elif defined(__powerpc64__) || defined(__aarch64__) - // On PowerPC64 we have two different address space layouts: 44- and 46-bit. - // We somehow need to figure out which one we are using now and choose - // one of 0x00000fffffffffffUL and 0x00003fffffffffffUL. - // Note that with 'ulimit -s unlimited' the stack is moved away from the top - // of the address space, so simply checking the stack address is not enough. - // This should (does) work for both PowerPC64 Endian modes. - // Similarly, aarch64 has multiple address space layouts: 39, 42 and 47-bit. - return (1ULL << (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1)) - 1; -# elif defined(__mips64) - return (1ULL << 40) - 1; // 0x000000ffffffffffUL; -# elif defined(__s390x__) - return (1ULL << 53) - 1; // 0x001fffffffffffffUL; -# else - return (1ULL << 47) - 1; // 0x00007fffffffffffUL; -# endif -#else // SANITIZER_WORDSIZE == 32 -# if defined(__s390__) - return (1ULL << 31) - 1; // 0x7fffffff; -# else - uptr res = (1ULL << 32) - 1; // 0xffffffff; - if (!common_flags()->full_address_space) - res -= GetKernelAreaSize(); - CHECK_LT(reinterpret_cast<uptr>(&res), res); - return res; -# endif -#endif // SANITIZER_WORDSIZE -} - void *MmapOrDie(uptr size, const char *mem_type, bool raw_report) { size = RoundUpTo(size, GetPageSizeCached()); uptr res = internal_mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); int reserrno; - if (internal_iserror(res, &reserrno)) + if (UNLIKELY(internal_iserror(res, &reserrno))) ReportMmapFailureAndDie(size, mem_type, "allocate", reserrno, raw_report); IncreaseTotalMmap(size); return (void *)res; @@ -137,7 +56,7 @@ void *MmapOrDie(uptr size, const char *mem_type, bool raw_report) { void UnmapOrDie(void *addr, uptr size) { if (!addr || !size) return; uptr res = internal_munmap(addr, size); - if (internal_iserror(res)) { + if (UNLIKELY(internal_iserror(res))) { Report("ERROR: %s failed to deallocate 0x%zx (%zd) bytes at address %p\n", SanitizerToolName, size, size, addr); CHECK("unable to unmap" && 0); @@ -145,21 +64,39 @@ void UnmapOrDie(void *addr, uptr size) { DecreaseTotalMmap(size); } +void *MmapOrDieOnFatalError(uptr size, const char *mem_type) { + size = RoundUpTo(size, GetPageSizeCached()); + uptr res = internal_mmap(nullptr, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANON, -1, 0); + int reserrno; + if (UNLIKELY(internal_iserror(res, &reserrno))) { + if (reserrno == ENOMEM) + return nullptr; + ReportMmapFailureAndDie(size, mem_type, "allocate", reserrno); + } + IncreaseTotalMmap(size); + return (void *)res; +} + // We want to map a chunk of address space aligned to 'alignment'. -// We do it by maping a bit more and then unmaping redundant pieces. +// We do it by mapping a bit more and then unmapping redundant pieces. // We probably can do it with fewer syscalls in some OS-dependent way. -void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type) { +void *MmapAlignedOrDieOnFatalError(uptr size, uptr alignment, + const char *mem_type) { CHECK(IsPowerOfTwo(size)); CHECK(IsPowerOfTwo(alignment)); uptr map_size = size + alignment; - uptr map_res = (uptr)MmapOrDie(map_size, mem_type); + uptr map_res = (uptr)MmapOrDieOnFatalError(map_size, mem_type); + if (UNLIKELY(!map_res)) + return nullptr; uptr map_end = map_res + map_size; uptr res = map_res; - if (res & (alignment - 1)) // Not aligned. - res = (map_res + alignment) & ~(alignment - 1); - uptr end = res + size; - if (res != map_res) + if (!IsAligned(res, alignment)) { + res = (map_res + alignment - 1) & ~(alignment - 1); UnmapOrDie((void*)map_res, res - map_res); + } + uptr end = res + size; if (end != map_end) UnmapOrDie((void*)end, map_end - end); return (void*)res; @@ -173,13 +110,13 @@ void *MmapNoReserveOrDie(uptr size, const char *mem_type) { MAP_PRIVATE | MAP_ANON | MAP_NORESERVE, -1, 0); int reserrno; - if (internal_iserror(p, &reserrno)) + if (UNLIKELY(internal_iserror(p, &reserrno))) ReportMmapFailureAndDie(size, mem_type, "allocate noreserve", reserrno); IncreaseTotalMmap(size); return (void *)p; } -void *MmapFixedOrDie(uptr fixed_addr, uptr size) { +void *MmapFixedImpl(uptr fixed_addr, uptr size, bool tolerate_enomem) { uptr PageSize = GetPageSizeCached(); uptr p = internal_mmap((void*)(fixed_addr & ~(PageSize - 1)), RoundUpTo(size, PageSize), @@ -187,8 +124,10 @@ void *MmapFixedOrDie(uptr fixed_addr, uptr size) { MAP_PRIVATE | MAP_ANON | MAP_FIXED, -1, 0); int reserrno; - if (internal_iserror(p, &reserrno)) { - char mem_type[30]; + if (UNLIKELY(internal_iserror(p, &reserrno))) { + if (tolerate_enomem && reserrno == ENOMEM) + return nullptr; + char mem_type[40]; internal_snprintf(mem_type, sizeof(mem_type), "memory at address 0x%zx", fixed_addr); ReportMmapFailureAndDie(size, mem_type, "allocate", reserrno); @@ -197,6 +136,14 @@ void *MmapFixedOrDie(uptr fixed_addr, uptr size) { return (void *)p; } +void *MmapFixedOrDie(uptr fixed_addr, uptr size) { + return MmapFixedImpl(fixed_addr, size, false /*tolerate_enomem*/); +} + +void *MmapFixedOrDieOnFatalError(uptr fixed_addr, uptr size) { + return MmapFixedImpl(fixed_addr, size, true /*tolerate_enomem*/); +} + bool MprotectNoAccess(uptr addr, uptr size) { return 0 == internal_mprotect((void*)addr, size, PROT_NONE); } @@ -284,13 +231,12 @@ static inline bool IntervalsAreSeparate(uptr start1, uptr end1, // memory). bool MemoryRangeIsAvailable(uptr range_start, uptr range_end) { MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr start, end; - while (proc_maps.Next(&start, &end, - /*offset*/nullptr, /*filename*/nullptr, - /*filename_size*/0, /*protection*/nullptr)) { - if (start == end) continue; // Empty range. - CHECK_NE(0, end); - if (!IntervalsAreSeparate(start, end - 1, range_start, range_end)) + MemoryMappedSegment segment; + while (proc_maps.Next(&segment)) { + if (segment.start == segment.end) continue; // Empty range. + CHECK_NE(0, segment.end); + if (!IntervalsAreSeparate(segment.start, segment.end - 1, range_start, + range_end)) return false; } return true; @@ -298,13 +244,13 @@ bool MemoryRangeIsAvailable(uptr range_start, uptr range_end) { void DumpProcessMap() { MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr start, end; const sptr kBufSize = 4095; char *filename = (char*)MmapOrDie(kBufSize, __func__); + MemoryMappedSegment segment(filename, kBufSize); Report("Process memory map follows:\n"); - while (proc_maps.Next(&start, &end, /* file_offset */nullptr, - filename, kBufSize, /* protection */nullptr)) { - Printf("\t%p-%p\t%s\n", (void*)start, (void*)end, filename); + while (proc_maps.Next(&segment)) { + Printf("\t%p-%p\t%s\n", (void *)segment.start, (void *)segment.end, + segment.filename); } Report("End of process memory map.\n"); UnmapOrDie(filename, kBufSize); @@ -334,14 +280,14 @@ void ReportFile::Write(const char *buffer, uptr length) { } bool GetCodeRangeForFile(const char *module, uptr *start, uptr *end) { - uptr s, e, off, prot; - InternalScopedString buff(kMaxPathLength); MemoryMappingLayout proc_maps(/*cache_enabled*/false); - while (proc_maps.Next(&s, &e, &off, buff.data(), buff.size(), &prot)) { - if ((prot & MemoryMappingLayout::kProtectionExecute) != 0 - && internal_strcmp(module, buff.data()) == 0) { - *start = s; - *end = e; + InternalScopedString buff(kMaxPathLength); + MemoryMappedSegment segment(buff.data(), kMaxPathLength); + while (proc_maps.Next(&segment)) { + if (segment.IsExecutable() && + internal_strcmp(module, segment.filename) == 0) { + *start = segment.start; + *end = segment.end; return true; } } @@ -358,6 +304,22 @@ SignalContext SignalContext::Create(void *siginfo, void *context) { return SignalContext(context, addr, pc, sp, bp, is_memory_access, write_flag); } +const char *DescribeSignalOrException(int signo) { + switch (signo) { + case SIGFPE: + return "FPE"; + case SIGILL: + return "ILL"; + case SIGABRT: + return "ABRT"; + case SIGSEGV: + return "SEGV"; + case SIGBUS: + return "BUS"; + } + return "UNKNOWN SIGNAL"; +} + } // namespace __sanitizer #endif // SANITIZER_POSIX diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.h index 7f862cd..e7d37cb 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix.h @@ -87,6 +87,9 @@ bool internal_sigismember(__sanitizer_sigset_t *set, int signum); uptr internal_execve(const char *filename, char *const argv[], char *const envp[]); + +bool IsStateDetached(int state); + } // namespace __sanitizer #endif // SANITIZER_POSIX_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc index dd62140..e113fb1 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc @@ -134,7 +134,8 @@ void SleepForMillis(int millis) { void Abort() { #if !SANITIZER_GO // If we are handling SIGABRT, unhandle it first. - if (IsHandledDeadlySignal(SIGABRT)) { + // TODO(vitalybuka): Check if handler belongs to sanitizer. + if (GetHandleSignalMode(SIGABRT) != kHandleSignalNo) { struct sigaction sigact; internal_memset(&sigact, 0, sizeof(sigact)); sigact.sa_sigaction = (sa_sigaction_t)SIG_DFL; @@ -188,8 +189,8 @@ void UnsetAlternateSignalStack() { static void MaybeInstallSigaction(int signum, SignalHandlerType handler) { - if (!IsHandledDeadlySignal(signum)) - return; + if (GetHandleSignalMode(signum) == kHandleSignalNo) return; + struct sigaction sigact; internal_memset(&sigact, 0, sizeof(sigact)); sigact.sa_sigaction = (sa_sigaction_t)handler; @@ -245,7 +246,6 @@ void PrepareForSandboxing(__sanitizer_sandbox_arguments *args) { // Same for /proc/self/exe in the symbolizer. #if !SANITIZER_GO Symbolizer::GetOrInit()->PrepareForSandboxing(); - CovPrepareForSandboxing(args); #endif } @@ -418,6 +418,10 @@ int WaitForProcess(pid_t pid) { return process_status; } +bool IsStateDetached(int state) { + return state == PTHREAD_CREATE_DETACHED; +} + } // namespace __sanitizer #endif // SANITIZER_POSIX diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_printf.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_printf.cc index c8317be..99b7ff1 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_printf.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_printf.cc @@ -214,15 +214,11 @@ void SetPrintfAndReportCallback(void (*callback)(const char *)) { } // Can be overriden in frontend. -#if SANITIZER_SUPPORTS_WEAK_HOOKS -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void OnPrint(const char *str) { - (void)str; -} -#elif SANITIZER_GO && defined(TSAN_EXTERNAL_HOOKS) -void OnPrint(const char *str); +#if SANITIZER_GO && defined(TSAN_EXTERNAL_HOOKS) +// Implementation must be defined in frontend. +extern "C" void OnPrint(const char *str); #else -void OnPrint(const char *str) { +SANITIZER_INTERFACE_WEAK_DEF(void, OnPrint, const char *str) { (void)str; } #endif diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps.h index 9dbb5ef..06d072b 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps.h @@ -31,13 +31,37 @@ struct ProcSelfMapsBuff { void ReadProcMaps(ProcSelfMapsBuff *proc_maps); #endif // SANITIZER_FREEBSD || SANITIZER_LINUX +// Memory protection masks. +static const uptr kProtectionRead = 1; +static const uptr kProtectionWrite = 2; +static const uptr kProtectionExecute = 4; +static const uptr kProtectionShared = 8; + +struct MemoryMappedSegment { + MemoryMappedSegment(char *buff = nullptr, uptr size = 0) + : filename(buff), filename_size(size) {} + ~MemoryMappedSegment() {} + + bool IsReadable() { return protection & kProtectionRead; } + bool IsWritable() { return protection & kProtectionWrite; } + bool IsExecutable() { return protection & kProtectionExecute; } + bool IsShared() { return protection & kProtectionShared; } + + uptr start; + uptr end; + uptr offset; + char *filename; // owned by caller + uptr filename_size; + uptr protection; + ModuleArch arch; + u8 uuid[kModuleUUIDSize]; +}; + class MemoryMappingLayout { public: explicit MemoryMappingLayout(bool cache_enabled); ~MemoryMappingLayout(); - bool Next(uptr *start, uptr *end, uptr *offset, char filename[], - uptr filename_size, uptr *protection, ModuleArch *arch = nullptr, - u8 *uuid = nullptr); + bool Next(MemoryMappedSegment *segment); void Reset(); // In some cases, e.g. when running under a sandbox on Linux, ASan is unable // to obtain the memory mappings. It should fall back to pre-cached data @@ -47,12 +71,6 @@ class MemoryMappingLayout { // Adds all mapped objects into a vector. void DumpListOfModules(InternalMmapVector<LoadedModule> *modules); - // Memory protection masks. - static const uptr kProtectionRead = 1; - static const uptr kProtectionWrite = 2; - static const uptr kProtectionExecute = 4; - static const uptr kProtectionShared = 8; - private: void LoadFromCache(); @@ -67,9 +85,7 @@ class MemoryMappingLayout { static StaticSpinMutex cache_lock_; // protects cached_proc_self_maps_. # elif SANITIZER_MAC template <u32 kLCSegment, typename SegmentCommand> - bool NextSegmentLoad(uptr *start, uptr *end, uptr *offset, char filename[], - uptr filename_size, ModuleArch *arch, u8 *uuid, - uptr *protection); + bool NextSegmentLoad(MemoryMappedSegment *segment); int current_image_; u32 current_magic_; u32 current_filetype_; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_common.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_common.cc index fac3fbd..b95f301 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_common.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_common.cc @@ -119,12 +119,10 @@ void MemoryMappingLayout::LoadFromCache() { void MemoryMappingLayout::DumpListOfModules( InternalMmapVector<LoadedModule> *modules) { Reset(); - uptr cur_beg, cur_end, cur_offset, prot; InternalScopedString module_name(kMaxPathLength); - for (uptr i = 0; Next(&cur_beg, &cur_end, &cur_offset, module_name.data(), - module_name.size(), &prot); - i++) { - const char *cur_name = module_name.data(); + MemoryMappedSegment segment(module_name.data(), module_name.size()); + for (uptr i = 0; Next(&segment); i++) { + const char *cur_name = segment.filename; if (cur_name[0] == '\0') continue; // Don't subtract 'cur_beg' from the first entry: @@ -138,10 +136,11 @@ void MemoryMappingLayout::DumpListOfModules( // mapped high at address space (in particular, higher than // shadow memory of the tool), so the module can't be the // first entry. - uptr base_address = (i ? cur_beg : 0) - cur_offset; + uptr base_address = (i ? segment.start : 0) - segment.offset; LoadedModule cur_module; cur_module.set(cur_name, base_address); - cur_module.addAddressRange(cur_beg, cur_end, prot & kProtectionExecute); + cur_module.addAddressRange(segment.start, segment.end, + segment.IsExecutable(), segment.IsWritable()); modules->push_back(cur_module); } } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_freebsd.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_freebsd.cc index 3021645..f0cdbeb 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_freebsd.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_freebsd.cc @@ -48,36 +48,27 @@ void ReadProcMaps(ProcSelfMapsBuff *proc_maps) { proc_maps->len = Size; } -bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, - char filename[], uptr filename_size, - uptr *protection, ModuleArch *arch, u8 *uuid) { - CHECK(!arch && "not implemented"); - CHECK(!uuid && "not implemented"); +bool MemoryMappingLayout::Next(MemoryMappedSegment *segment) { char *last = proc_self_maps_.data + proc_self_maps_.len; if (current_ >= last) return false; - uptr dummy; - if (!start) start = &dummy; - if (!end) end = &dummy; - if (!offset) offset = &dummy; - if (!protection) protection = &dummy; struct kinfo_vmentry *VmEntry = (struct kinfo_vmentry*)current_; - *start = (uptr)VmEntry->kve_start; - *end = (uptr)VmEntry->kve_end; - *offset = (uptr)VmEntry->kve_offset; + segment->start = (uptr)VmEntry->kve_start; + segment->end = (uptr)VmEntry->kve_end; + segment->offset = (uptr)VmEntry->kve_offset; - *protection = 0; + segment->protection = 0; if ((VmEntry->kve_protection & KVME_PROT_READ) != 0) - *protection |= kProtectionRead; + segment->protection |= kProtectionRead; if ((VmEntry->kve_protection & KVME_PROT_WRITE) != 0) - *protection |= kProtectionWrite; + segment->protection |= kProtectionWrite; if ((VmEntry->kve_protection & KVME_PROT_EXEC) != 0) - *protection |= kProtectionExecute; + segment->protection |= kProtectionExecute; - if (filename != NULL && filename_size > 0) { - internal_snprintf(filename, - Min(filename_size, (uptr)PATH_MAX), - "%s", VmEntry->kve_path); + if (segment->filename != NULL && segment->filename_size > 0) { + internal_snprintf(segment->filename, + Min(segment->filename_size, (uptr)PATH_MAX), "%s", + VmEntry->kve_path); } current_ += VmEntry->kve_structsize; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_linux.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_linux.cc index fdf85b7..1bcad2b 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_linux.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_linux.cc @@ -18,49 +18,36 @@ namespace __sanitizer { void ReadProcMaps(ProcSelfMapsBuff *proc_maps) { - CHECK(ReadFileToBuffer("/proc/self/maps", &proc_maps->data, - &proc_maps->mmaped_size, &proc_maps->len)); + ReadFileToBuffer("/proc/self/maps", &proc_maps->data, &proc_maps->mmaped_size, + &proc_maps->len); } static bool IsOneOf(char c, char c1, char c2) { return c == c1 || c == c2; } -bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, - char filename[], uptr filename_size, - uptr *protection, ModuleArch *arch, u8 *uuid) { - CHECK(!arch && "not implemented"); - CHECK(!uuid && "not implemented"); +bool MemoryMappingLayout::Next(MemoryMappedSegment *segment) { char *last = proc_self_maps_.data + proc_self_maps_.len; if (current_ >= last) return false; - uptr dummy; - if (!start) start = &dummy; - if (!end) end = &dummy; - if (!offset) offset = &dummy; - if (!protection) protection = &dummy; char *next_line = (char*)internal_memchr(current_, '\n', last - current_); if (next_line == 0) next_line = last; // Example: 08048000-08056000 r-xp 00000000 03:0c 64593 /foo/bar - *start = ParseHex(¤t_); + segment->start = ParseHex(¤t_); CHECK_EQ(*current_++, '-'); - *end = ParseHex(¤t_); + segment->end = ParseHex(¤t_); CHECK_EQ(*current_++, ' '); CHECK(IsOneOf(*current_, '-', 'r')); - *protection = 0; - if (*current_++ == 'r') - *protection |= kProtectionRead; + segment->protection = 0; + if (*current_++ == 'r') segment->protection |= kProtectionRead; CHECK(IsOneOf(*current_, '-', 'w')); - if (*current_++ == 'w') - *protection |= kProtectionWrite; + if (*current_++ == 'w') segment->protection |= kProtectionWrite; CHECK(IsOneOf(*current_, '-', 'x')); - if (*current_++ == 'x') - *protection |= kProtectionExecute; + if (*current_++ == 'x') segment->protection |= kProtectionExecute; CHECK(IsOneOf(*current_, 's', 'p')); - if (*current_++ == 's') - *protection |= kProtectionShared; + if (*current_++ == 's') segment->protection |= kProtectionShared; CHECK_EQ(*current_++, ' '); - *offset = ParseHex(¤t_); + segment->offset = ParseHex(¤t_); CHECK_EQ(*current_++, ' '); ParseHex(¤t_); CHECK_EQ(*current_++, ':'); @@ -75,14 +62,12 @@ bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, while (current_ < next_line && *current_ == ' ') current_++; // Fill in the filename. - uptr i = 0; - while (current_ < next_line) { - if (filename && i < filename_size - 1) - filename[i++] = *current_; - current_++; + if (segment->filename) { + uptr len = Min((uptr)(next_line - current_), segment->filename_size - 1); + internal_strncpy(segment->filename, current_, len); + segment->filename[len] = 0; } - if (filename && i < filename_size) - filename[i] = 0; + current_ = next_line + 1; return true; } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cc index 2831f28..560451a 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cc @@ -18,6 +18,7 @@ #include <mach-o/dyld.h> #include <mach-o/loader.h> +#include <mach/mach.h> // These are not available in older macOS SDKs. #ifndef CPU_SUBTYPE_X86_64_H @@ -71,6 +72,13 @@ void MemoryMappingLayout::Reset() { internal_memset(current_uuid_, 0, kModuleUUIDSize); } +// The dyld load address should be unchanged throughout process execution, +// and it is expensive to compute once many libraries have been loaded, +// so cache it here and do not reset. +static mach_header *dyld_hdr = 0; +static const char kDyldPath[] = "/usr/lib/dyld"; +static const int kDyldImageIdx = -1; + // static void MemoryMappingLayout::CacheMemoryMappings() { // No-op on Mac for now. @@ -80,6 +88,48 @@ void MemoryMappingLayout::LoadFromCache() { // No-op on Mac for now. } +// _dyld_get_image_header() and related APIs don't report dyld itself. +// We work around this by manually recursing through the memory map +// until we hit a Mach header matching dyld instead. These recurse +// calls are expensive, but the first memory map generation occurs +// early in the process, when dyld is one of the only images loaded, +// so it will be hit after only a few iterations. +static mach_header *get_dyld_image_header() { + mach_port_name_t port; + if (task_for_pid(mach_task_self(), internal_getpid(), &port) != + KERN_SUCCESS) { + return nullptr; + } + + unsigned depth = 1; + vm_size_t size = 0; + vm_address_t address = 0; + kern_return_t err = KERN_SUCCESS; + mach_msg_type_number_t count = VM_REGION_SUBMAP_INFO_COUNT_64; + + while (true) { + struct vm_region_submap_info_64 info; + err = vm_region_recurse_64(port, &address, &size, &depth, + (vm_region_info_t)&info, &count); + if (err != KERN_SUCCESS) return nullptr; + + if (size >= sizeof(mach_header) && info.protection & kProtectionRead) { + mach_header *hdr = (mach_header *)address; + if ((hdr->magic == MH_MAGIC || hdr->magic == MH_MAGIC_64) && + hdr->filetype == MH_DYLINKER) { + return hdr; + } + } + address += size; + } +} + +const mach_header *get_dyld_hdr() { + if (!dyld_hdr) dyld_hdr = get_dyld_image_header(); + + return dyld_hdr; +} + // Next and NextSegmentLoad were inspired by base/sysinfo.cc in // Google Perftools, https://github.com/gperftools/gperftools. @@ -88,38 +138,39 @@ void MemoryMappingLayout::LoadFromCache() { // segment. // Note that the segment addresses are not necessarily sorted. template <u32 kLCSegment, typename SegmentCommand> -bool MemoryMappingLayout::NextSegmentLoad(uptr *start, uptr *end, uptr *offset, - char filename[], uptr filename_size, - ModuleArch *arch, u8 *uuid, - uptr *protection) { +bool MemoryMappingLayout::NextSegmentLoad(MemoryMappedSegment *segment) { const char *lc = current_load_cmd_addr_; current_load_cmd_addr_ += ((const load_command *)lc)->cmdsize; if (((const load_command *)lc)->cmd == kLCSegment) { - const sptr dlloff = _dyld_get_image_vmaddr_slide(current_image_); const SegmentCommand* sc = (const SegmentCommand *)lc; - if (start) *start = sc->vmaddr + dlloff; - if (protection) { - // Return the initial protection. - *protection = sc->initprot; - } - if (end) *end = sc->vmaddr + sc->vmsize + dlloff; - if (offset) { - if (current_filetype_ == /*MH_EXECUTE*/ 0x2) { - *offset = sc->vmaddr; - } else { - *offset = sc->fileoff; - } - } - if (filename) { - internal_strncpy(filename, _dyld_get_image_name(current_image_), - filename_size); - } - if (arch) { - *arch = current_arch_; + + if (current_image_ == kDyldImageIdx) { + // vmaddr is masked with 0xfffff because on macOS versions < 10.12, + // it contains an absolute address rather than an offset for dyld. + // To make matters even more complicated, this absolute address + // isn't actually the absolute segment address, but the offset portion + // of the address is accurate when combined with the dyld base address, + // and the mask will give just this offset. + segment->start = (sc->vmaddr & 0xfffff) + (uptr)get_dyld_hdr(); + segment->end = (sc->vmaddr & 0xfffff) + sc->vmsize + (uptr)get_dyld_hdr(); + } else { + const sptr dlloff = _dyld_get_image_vmaddr_slide(current_image_); + segment->start = sc->vmaddr + dlloff; + segment->end = sc->vmaddr + sc->vmsize + dlloff; } - if (uuid) { - internal_memcpy(uuid, current_uuid_, kModuleUUIDSize); + + // Return the initial protection. + segment->protection = sc->initprot; + segment->offset = + (current_filetype_ == /*MH_EXECUTE*/ 0x2) ? sc->vmaddr : sc->fileoff; + if (segment->filename) { + const char *src = (current_image_ == kDyldImageIdx) + ? kDyldPath + : _dyld_get_image_name(current_image_); + internal_strncpy(segment->filename, src, segment->filename_size); } + segment->arch = current_arch_; + internal_memcpy(segment->uuid, current_uuid_, kModuleUUIDSize); return true; } return false; @@ -180,11 +231,11 @@ static bool IsModuleInstrumented(const load_command *first_lc) { return false; } -bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, - char filename[], uptr filename_size, - uptr *protection, ModuleArch *arch, u8 *uuid) { - for (; current_image_ >= 0; current_image_--) { - const mach_header* hdr = _dyld_get_image_header(current_image_); +bool MemoryMappingLayout::Next(MemoryMappedSegment *segment) { + for (; current_image_ >= kDyldImageIdx; current_image_--) { + const mach_header *hdr = (current_image_ == kDyldImageIdx) + ? get_dyld_hdr() + : _dyld_get_image_header(current_image_); if (!hdr) continue; if (current_load_cmd_count_ < 0) { // Set up for this image; @@ -218,16 +269,13 @@ bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, #ifdef MH_MAGIC_64 case MH_MAGIC_64: { if (NextSegmentLoad<LC_SEGMENT_64, struct segment_command_64>( - start, end, offset, filename, filename_size, arch, uuid, - protection)) + segment)) return true; break; } #endif case MH_MAGIC: { - if (NextSegmentLoad<LC_SEGMENT, struct segment_command>( - start, end, offset, filename, filename_size, arch, uuid, - protection)) + if (NextSegmentLoad<LC_SEGMENT, struct segment_command>(segment)) return true; break; } @@ -242,27 +290,22 @@ bool MemoryMappingLayout::Next(uptr *start, uptr *end, uptr *offset, void MemoryMappingLayout::DumpListOfModules( InternalMmapVector<LoadedModule> *modules) { Reset(); - uptr cur_beg, cur_end, prot; - ModuleArch cur_arch; - u8 cur_uuid[kModuleUUIDSize]; InternalScopedString module_name(kMaxPathLength); - for (uptr i = 0; Next(&cur_beg, &cur_end, 0, module_name.data(), - module_name.size(), &prot, &cur_arch, &cur_uuid[0]); - i++) { - const char *cur_name = module_name.data(); - if (cur_name[0] == '\0') - continue; + MemoryMappedSegment segment(module_name.data(), kMaxPathLength); + for (uptr i = 0; Next(&segment); i++) { + if (segment.filename[0] == '\0') continue; LoadedModule *cur_module = nullptr; if (!modules->empty() && - 0 == internal_strcmp(cur_name, modules->back().full_name())) { + 0 == internal_strcmp(segment.filename, modules->back().full_name())) { cur_module = &modules->back(); } else { modules->push_back(LoadedModule()); cur_module = &modules->back(); - cur_module->set(cur_name, cur_beg, cur_arch, cur_uuid, - current_instrumented_); + cur_module->set(segment.filename, segment.start, segment.arch, + segment.uuid, current_instrumented_); } - cur_module->addAddressRange(cur_beg, cur_end, prot & kProtectionExecute); + cur_module->addAddressRange(segment.start, segment.end, + segment.IsExecutable(), segment.IsWritable()); } } diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_quarantine.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_quarantine.h index 1a0d954..db38867 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_quarantine.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_quarantine.h @@ -31,6 +31,40 @@ struct QuarantineBatch { uptr size; uptr count; void *batch[kSize]; + + void init(void *ptr, uptr size) { + count = 1; + batch[0] = ptr; + this->size = size + sizeof(QuarantineBatch); // Account for the batch size. + } + + // The total size of quarantined nodes recorded in this batch. + uptr quarantined_size() const { + return size - sizeof(QuarantineBatch); + } + + void push_back(void *ptr, uptr size) { + CHECK_LT(count, kSize); + batch[count++] = ptr; + this->size += size; + } + + bool can_merge(const QuarantineBatch* const from) const { + return count + from->count <= kSize; + } + + void merge(QuarantineBatch* const from) { + CHECK_LE(count + from->count, kSize); + CHECK_GE(size, sizeof(QuarantineBatch)); + + for (uptr i = 0; i < from->count; ++i) + batch[count + i] = from->batch[i]; + count += from->count; + size += from->quarantined_size(); + + from->count = 0; + from->size = sizeof(QuarantineBatch); + } }; COMPILER_CHECK(sizeof(QuarantineBatch) <= (1 << 13)); // 8Kb. @@ -69,7 +103,7 @@ class Quarantine { if (cache_size) { c->Enqueue(cb, ptr, size); } else { - // cache_size == 0 only when size == 0 (see Init). + // GetCacheSize() == 0 only when GetSize() == 0 (see Init). cb.Recycle(ptr); } // Check cache size anyway to accommodate for runtime cache_size change. @@ -88,6 +122,8 @@ class Quarantine { void PrintStats() const { // It assumes that the world is stopped, just as the allocator's PrintStats. + Printf("Quarantine limits: global: %zdMb; thread local: %zdKb\n", + GetSize() >> 20, GetCacheSize() >> 10); cache_.PrintStats(); } @@ -108,9 +144,27 @@ class Quarantine { uptr min_size = atomic_load(&min_size_, memory_order_relaxed); { SpinMutexLock l(&cache_mutex_); + // Go over the batches and merge partially filled ones to + // save some memory, otherwise batches themselves (since the memory used + // by them is counted against quarantine limit) can overcome the actual + // user's quarantined chunks, which diminishes the purpose of the + // quarantine. + uptr cache_size = cache_.Size(); + uptr overhead_size = cache_.OverheadSize(); + CHECK_GE(cache_size, overhead_size); + // Do the merge only when overhead exceeds this predefined limit (might + // require some tuning). It saves us merge attempt when the batch list + // quarantine is unlikely to contain batches suitable for merge. + const uptr kOverheadThresholdPercents = 100; + if (cache_size > overhead_size && + overhead_size * (100 + kOverheadThresholdPercents) > + cache_size * kOverheadThresholdPercents) { + cache_.MergeBatches(&tmp); + } + // Extract enough chunks from the quarantine to get below the max + // quarantine size and leave some leeway for the newly quarantined chunks. while (cache_.Size() > min_size) { - QuarantineBatch *b = cache_.DequeueBatch(); - tmp.EnqueueBatch(b); + tmp.EnqueueBatch(cache_.DequeueBatch()); } } recycle_mutex_.Unlock(); @@ -145,26 +199,33 @@ class QuarantineCache { list_.clear(); } + // Total memory used, including internal accounting. uptr Size() const { return atomic_load(&size_, memory_order_relaxed); } + // Memory used for internal accounting. + uptr OverheadSize() const { + return list_.size() * sizeof(QuarantineBatch); + } + void Enqueue(Callback cb, void *ptr, uptr size) { if (list_.empty() || list_.back()->count == QuarantineBatch::kSize) { - AllocBatch(cb); - size += sizeof(QuarantineBatch); // Count the batch in Quarantine size. + QuarantineBatch *b = (QuarantineBatch *)cb.Allocate(sizeof(*b)); + CHECK(b); + b->init(ptr, size); + EnqueueBatch(b); + } else { + list_.back()->push_back(ptr, size); + SizeAdd(size); } - QuarantineBatch *b = list_.back(); - CHECK(b); - b->batch[b->count++] = ptr; - b->size += size; - SizeAdd(size); } - void Transfer(QuarantineCache *c) { - list_.append_back(&c->list_); - SizeAdd(c->Size()); - atomic_store(&c->size_, 0, memory_order_relaxed); + void Transfer(QuarantineCache *from_cache) { + list_.append_back(&from_cache->list_); + SizeAdd(from_cache->Size()); + + atomic_store(&from_cache->size_, 0, memory_order_relaxed); } void EnqueueBatch(QuarantineBatch *b) { @@ -181,19 +242,51 @@ class QuarantineCache { return b; } + void MergeBatches(QuarantineCache *to_deallocate) { + uptr extracted_size = 0; + QuarantineBatch *current = list_.front(); + while (current && current->next) { + if (current->can_merge(current->next)) { + QuarantineBatch *extracted = current->next; + // Move all the chunks into the current batch. + current->merge(extracted); + CHECK_EQ(extracted->count, 0); + CHECK_EQ(extracted->size, sizeof(QuarantineBatch)); + // Remove the next batch from the list and account for its size. + list_.extract(current, extracted); + extracted_size += extracted->size; + // Add it to deallocation list. + to_deallocate->EnqueueBatch(extracted); + } else { + current = current->next; + } + } + SizeSub(extracted_size); + } + void PrintStats() const { uptr batch_count = 0; - uptr total_quarantine_bytes = 0; + uptr total_overhead_bytes = 0; + uptr total_bytes = 0; uptr total_quarantine_chunks = 0; for (List::ConstIterator it = list_.begin(); it != list_.end(); ++it) { batch_count++; - total_quarantine_bytes += (*it).size; + total_bytes += (*it).size; + total_overhead_bytes += (*it).size - (*it).quarantined_size(); total_quarantine_chunks += (*it).count; } - Printf("Global quarantine stats: batches: %zd; bytes: %zd; chunks: %zd " - "(capacity: %zd chunks)\n", - batch_count, total_quarantine_bytes, total_quarantine_chunks, - batch_count * QuarantineBatch::kSize); + uptr quarantine_chunks_capacity = batch_count * QuarantineBatch::kSize; + int chunks_usage_percent = quarantine_chunks_capacity == 0 ? + 0 : total_quarantine_chunks * 100 / quarantine_chunks_capacity; + uptr total_quarantined_bytes = total_bytes - total_overhead_bytes; + int memory_overhead_percent = total_quarantined_bytes == 0 ? + 0 : total_overhead_bytes * 100 / total_quarantined_bytes; + Printf("Global quarantine stats: batches: %zd; bytes: %zd (user: %zd); " + "chunks: %zd (capacity: %zd); %d%% chunks used; %d%% memory overhead" + "\n", + batch_count, total_bytes, total_quarantined_bytes, + total_quarantine_chunks, quarantine_chunks_capacity, + chunks_usage_percent, memory_overhead_percent); } private: @@ -208,15 +301,6 @@ class QuarantineCache { void SizeSub(uptr sub) { atomic_store(&size_, Size() - sub, memory_order_relaxed); } - - NOINLINE QuarantineBatch* AllocBatch(Callback cb) { - QuarantineBatch *b = (QuarantineBatch *)cb.Allocate(sizeof(*b)); - CHECK(b); - b->count = 0; - b->size = 0; - list_.push_back(b); - return b; - } }; } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc index 36c98d0..747a4a7 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc @@ -43,7 +43,8 @@ void StackTrace::Print() const { if (dedup_frames-- > 0) { if (dedup_token.length()) dedup_token.append("--"); - dedup_token.append(cur->info.function); + if (cur->info.function != nullptr) + dedup_token.append(cur->info.function); } } frames->ClearAll(); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld.h index aa6f5d8..20b49ae7 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld.h @@ -18,36 +18,32 @@ #include "sanitizer_common.h" namespace __sanitizer { -typedef int SuspendedThreadID; + +enum PtraceRegistersStatus { + REGISTERS_UNAVAILABLE_FATAL = -1, + REGISTERS_UNAVAILABLE = 0, + REGISTERS_AVAILABLE = 1 +}; // Holds the list of suspended threads and provides an interface to dump their // register contexts. class SuspendedThreadsList { public: - SuspendedThreadsList() - : thread_ids_(1024) {} - SuspendedThreadID GetThreadID(uptr index) const { - CHECK_LT(index, thread_ids_.size()); - return thread_ids_[index]; + SuspendedThreadsList() = default; + + // Can't declare pure virtual functions in sanitizer runtimes: + // __cxa_pure_virtual might be unavailable. Use UNIMPLEMENTED() instead. + virtual PtraceRegistersStatus GetRegistersAndSP(uptr index, uptr *buffer, + uptr *sp) const { + UNIMPLEMENTED(); } - int GetRegistersAndSP(uptr index, uptr *buffer, uptr *sp) const; + // The buffer in GetRegistersAndSP should be at least this big. - static uptr RegisterCount(); - uptr thread_count() const { return thread_ids_.size(); } - bool Contains(SuspendedThreadID thread_id) const { - for (uptr i = 0; i < thread_ids_.size(); i++) { - if (thread_ids_[i] == thread_id) - return true; - } - return false; - } - void Append(SuspendedThreadID thread_id) { - thread_ids_.push_back(thread_id); - } + virtual uptr RegisterCount() const { UNIMPLEMENTED(); } + virtual uptr ThreadCount() const { UNIMPLEMENTED(); } + virtual tid_t GetThreadID(uptr index) const { UNIMPLEMENTED(); } private: - InternalMmapVector<SuspendedThreadID> thread_ids_; - // Prohibit copy and assign. SuspendedThreadsList(const SuspendedThreadsList&); void operator=(const SuspendedThreadsList&); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cc index eb4c403..d7fa5f6 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cc @@ -16,7 +16,8 @@ #if SANITIZER_LINUX && (defined(__x86_64__) || defined(__mips__) || \ defined(__aarch64__) || defined(__powerpc64__) || \ - defined(__s390__)) + defined(__s390__) || defined(__i386__) || \ + defined(__arm__)) #include "sanitizer_stoptheworld.h" @@ -31,17 +32,13 @@ #include <sys/types.h> // for pid_t #include <sys/uio.h> // for iovec #include <elf.h> // for NT_PRSTATUS -#if SANITIZER_ANDROID && defined(__arm__) -# include <linux/user.h> // for pt_regs -#else -# ifdef __aarch64__ +#if defined(__aarch64__) && !SANITIZER_ANDROID // GLIBC 2.20+ sys/user does not include asm/ptrace.h -# include <asm/ptrace.h> -# endif -# include <sys/user.h> // for user_regs_struct -# if SANITIZER_ANDROID && SANITIZER_MIPS -# include <asm/reg.h> // for mips SP register in sys/user.h -# endif +# include <asm/ptrace.h> +#endif +#include <sys/user.h> // for user_regs_struct +#if SANITIZER_ANDROID && SANITIZER_MIPS +# include <asm/reg.h> // for mips SP register in sys/user.h #endif #include <sys/wait.h> // for signal-related stuff @@ -81,7 +78,22 @@ namespace __sanitizer { -COMPILER_CHECK(sizeof(SuspendedThreadID) == sizeof(pid_t)); +class SuspendedThreadsListLinux : public SuspendedThreadsList { + public: + SuspendedThreadsListLinux() : thread_ids_(1024) {} + + tid_t GetThreadID(uptr index) const; + uptr ThreadCount() const; + bool ContainsTid(tid_t thread_id) const; + void Append(tid_t tid); + + PtraceRegistersStatus GetRegistersAndSP(uptr index, uptr *buffer, + uptr *sp) const; + uptr RegisterCount() const; + + private: + InternalMmapVector<tid_t> thread_ids_; +}; // Structure for passing arguments into the tracer thread. struct TracerThreadArgument { @@ -106,31 +118,31 @@ class ThreadSuspender { bool SuspendAllThreads(); void ResumeAllThreads(); void KillAllThreads(); - SuspendedThreadsList &suspended_threads_list() { + SuspendedThreadsListLinux &suspended_threads_list() { return suspended_threads_list_; } TracerThreadArgument *arg; private: - SuspendedThreadsList suspended_threads_list_; + SuspendedThreadsListLinux suspended_threads_list_; pid_t pid_; - bool SuspendThread(SuspendedThreadID thread_id); + bool SuspendThread(tid_t thread_id); }; -bool ThreadSuspender::SuspendThread(SuspendedThreadID tid) { +bool ThreadSuspender::SuspendThread(tid_t tid) { // Are we already attached to this thread? // Currently this check takes linear time, however the number of threads is // usually small. - if (suspended_threads_list_.Contains(tid)) - return false; + if (suspended_threads_list_.ContainsTid(tid)) return false; int pterrno; if (internal_iserror(internal_ptrace(PTRACE_ATTACH, tid, nullptr, nullptr), &pterrno)) { // Either the thread is dead, or something prevented us from attaching. // Log this event and move on. - VReport(1, "Could not attach to thread %d (errno %d).\n", tid, pterrno); + VReport(1, "Could not attach to thread %zu (errno %d).\n", (uptr)tid, + pterrno); return false; } else { - VReport(2, "Attached to thread %d.\n", tid); + VReport(2, "Attached to thread %zu.\n", (uptr)tid); // The thread is not guaranteed to stop before ptrace returns, so we must // wait on it. Note: if the thread receives a signal concurrently, // we can get notification about the signal before notification about stop. @@ -148,8 +160,8 @@ bool ThreadSuspender::SuspendThread(SuspendedThreadID tid) { if (internal_iserror(waitpid_status, &wperrno)) { // Got a ECHILD error. I don't think this situation is possible, but it // doesn't hurt to report it. - VReport(1, "Waiting on thread %d failed, detaching (errno %d).\n", - tid, wperrno); + VReport(1, "Waiting on thread %zu failed, detaching (errno %d).\n", + (uptr)tid, wperrno); internal_ptrace(PTRACE_DETACH, tid, nullptr, nullptr); return false; } @@ -166,7 +178,7 @@ bool ThreadSuspender::SuspendThread(SuspendedThreadID tid) { } void ThreadSuspender::ResumeAllThreads() { - for (uptr i = 0; i < suspended_threads_list_.thread_count(); i++) { + for (uptr i = 0; i < suspended_threads_list_.ThreadCount(); i++) { pid_t tid = suspended_threads_list_.GetThreadID(i); int pterrno; if (!internal_iserror(internal_ptrace(PTRACE_DETACH, tid, nullptr, nullptr), @@ -182,7 +194,7 @@ void ThreadSuspender::ResumeAllThreads() { } void ThreadSuspender::KillAllThreads() { - for (uptr i = 0; i < suspended_threads_list_.thread_count(); i++) + for (uptr i = 0; i < suspended_threads_list_.ThreadCount(); i++) internal_ptrace(PTRACE_KILL, suspended_threads_list_.GetThreadID(i), nullptr, nullptr); } @@ -275,7 +287,7 @@ static int TracerThread(void* argument) { // Alternate stack for signal handling. InternalScopedBuffer<char> handler_stack_memory(kHandlerStackSize); - struct sigaltstack handler_stack; + stack_t handler_stack; internal_memset(&handler_stack, 0, sizeof(handler_stack)); handler_stack.ss_sp = handler_stack_memory.data(); handler_stack.ss_size = kHandlerStackSize; @@ -493,9 +505,28 @@ typedef _user_regs_struct regs_struct; #error "Unsupported architecture" #endif // SANITIZER_ANDROID && defined(__arm__) -int SuspendedThreadsList::GetRegistersAndSP(uptr index, - uptr *buffer, - uptr *sp) const { +tid_t SuspendedThreadsListLinux::GetThreadID(uptr index) const { + CHECK_LT(index, thread_ids_.size()); + return thread_ids_[index]; +} + +uptr SuspendedThreadsListLinux::ThreadCount() const { + return thread_ids_.size(); +} + +bool SuspendedThreadsListLinux::ContainsTid(tid_t thread_id) const { + for (uptr i = 0; i < thread_ids_.size(); i++) { + if (thread_ids_[i] == thread_id) return true; + } + return false; +} + +void SuspendedThreadsListLinux::Append(tid_t tid) { + thread_ids_.push_back(tid); +} + +PtraceRegistersStatus SuspendedThreadsListLinux::GetRegistersAndSP( + uptr index, uptr *buffer, uptr *sp) const { pid_t tid = GetThreadID(index); regs_struct regs; int pterrno; @@ -513,19 +544,23 @@ int SuspendedThreadsList::GetRegistersAndSP(uptr index, if (isErr) { VReport(1, "Could not get registers from thread %d (errno %d).\n", tid, pterrno); - return -1; + // ESRCH means that the given thread is not suspended or already dead. + // Therefore it's unsafe to inspect its data (e.g. walk through stack) and + // we should notify caller about this. + return pterrno == ESRCH ? REGISTERS_UNAVAILABLE_FATAL + : REGISTERS_UNAVAILABLE; } *sp = regs.REG_SP; internal_memcpy(buffer, ®s, sizeof(regs)); - return 0; + return REGISTERS_AVAILABLE; } -uptr SuspendedThreadsList::RegisterCount() { +uptr SuspendedThreadsListLinux::RegisterCount() const { return sizeof(regs_struct) / sizeof(uptr); } } // namespace __sanitizer #endif // SANITIZER_LINUX && (defined(__x86_64__) || defined(__mips__) // || defined(__aarch64__) || defined(__powerpc64__) - // || defined(__s390__) + // || defined(__s390__) || defined(__i386__) || defined(__arm__) diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_mac.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_mac.cc new file mode 100644 index 0000000..0c27c47 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_mac.cc @@ -0,0 +1,186 @@ +//===-- sanitizer_stoptheworld_mac.cc -------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// See sanitizer_stoptheworld.h for details. +// +//===----------------------------------------------------------------------===// + +#include "sanitizer_platform.h" + +#if SANITIZER_MAC && (defined(__x86_64__) || defined(__aarch64__) || \ + defined(__i386)) + +#include <mach/mach.h> +#include <mach/thread_info.h> +#include <pthread.h> + +#include "sanitizer_stoptheworld.h" + +namespace __sanitizer { +typedef struct { + tid_t tid; + thread_t thread; +} SuspendedThreadInfo; + +class SuspendedThreadsListMac : public SuspendedThreadsList { + public: + SuspendedThreadsListMac() : threads_(1024) {} + + tid_t GetThreadID(uptr index) const; + thread_t GetThread(uptr index) const; + uptr ThreadCount() const; + bool ContainsThread(thread_t thread) const; + void Append(thread_t thread); + + PtraceRegistersStatus GetRegistersAndSP(uptr index, uptr *buffer, + uptr *sp) const; + uptr RegisterCount() const; + + private: + InternalMmapVector<SuspendedThreadInfo> threads_; +}; + +struct RunThreadArgs { + StopTheWorldCallback callback; + void *argument; +}; + +void RunThread(void *arg) { + struct RunThreadArgs *run_args = (struct RunThreadArgs *)arg; + SuspendedThreadsListMac suspended_threads_list; + + mach_port_t task; + kern_return_t err = task_for_pid(mach_task_self(), internal_getpid(), &task); + if (err != KERN_SUCCESS) { + VReport(1, "Getting task from pid failed (errno %d).\n", err); + return; + } + + thread_array_t threads; + mach_msg_type_number_t num_threads; + + err = task_threads(task, &threads, &num_threads); + if (err != KERN_SUCCESS) { + VReport(1, "Failed to get threads for task (errno %d).\n", err); + return; + } + + thread_t thread_self = mach_thread_self(); + for (unsigned int i = 0; i < num_threads; ++i) { + if (threads[i] == thread_self) continue; + + thread_suspend(threads[i]); + suspended_threads_list.Append(threads[i]); + } + + run_args->callback(suspended_threads_list, run_args->argument); + + uptr num_suspended = suspended_threads_list.ThreadCount(); + for (unsigned int i = 0; i < num_suspended; ++i) { + thread_resume(suspended_threads_list.GetThread(i)); + } +} + +void StopTheWorld(StopTheWorldCallback callback, void *argument) { + struct RunThreadArgs arg = {callback, argument}; + pthread_t run_thread = (pthread_t)internal_start_thread(RunThread, &arg); + internal_join_thread(run_thread); +} + +#if defined(__x86_64__) +typedef x86_thread_state64_t regs_struct; + +#define SP_REG __rsp + +#elif defined(__aarch64__) +typedef arm_thread_state64_t regs_struct; + +# if __DARWIN_UNIX03 +# define SP_REG __sp +# else +# define SP_REG sp +# endif + +#elif defined(__i386) +typedef x86_thread_state32_t regs_struct; + +#define SP_REG __esp + +#else +#error "Unsupported architecture" +#endif + +tid_t SuspendedThreadsListMac::GetThreadID(uptr index) const { + CHECK_LT(index, threads_.size()); + return threads_[index].tid; +} + +thread_t SuspendedThreadsListMac::GetThread(uptr index) const { + CHECK_LT(index, threads_.size()); + return threads_[index].thread; +} + +uptr SuspendedThreadsListMac::ThreadCount() const { + return threads_.size(); +} + +bool SuspendedThreadsListMac::ContainsThread(thread_t thread) const { + for (uptr i = 0; i < threads_.size(); i++) { + if (threads_[i].thread == thread) return true; + } + return false; +} + +void SuspendedThreadsListMac::Append(thread_t thread) { + thread_identifier_info_data_t info; + mach_msg_type_number_t info_count = THREAD_IDENTIFIER_INFO_COUNT; + kern_return_t err = thread_info(thread, THREAD_IDENTIFIER_INFO, + (thread_info_t)&info, &info_count); + if (err != KERN_SUCCESS) { + VReport(1, "Error - unable to get thread ident for a thread\n"); + return; + } + threads_.push_back({info.thread_id, thread}); +} + +PtraceRegistersStatus SuspendedThreadsListMac::GetRegistersAndSP( + uptr index, uptr *buffer, uptr *sp) const { + thread_t thread = GetThread(index); + regs_struct regs; + int err; + mach_msg_type_number_t reg_count = MACHINE_THREAD_STATE_COUNT; + err = thread_get_state(thread, MACHINE_THREAD_STATE, (thread_state_t)®s, + ®_count); + if (err != KERN_SUCCESS) { + VReport(1, "Error - unable to get registers for a thread\n"); + // KERN_INVALID_ARGUMENT indicates that either the flavor is invalid, + // or the thread does not exist. The other possible error case, + // MIG_ARRAY_TOO_LARGE, means that the state is too large, but it's + // still safe to proceed. + return err == KERN_INVALID_ARGUMENT ? REGISTERS_UNAVAILABLE_FATAL + : REGISTERS_UNAVAILABLE; + } + + internal_memcpy(buffer, ®s, sizeof(regs)); + *sp = regs.SP_REG; + + // On x86_64 and aarch64, we must account for the stack redzone, which is 128 + // bytes. + if (SANITIZER_WORDSIZE == 64) *sp -= 128; + + return REGISTERS_AVAILABLE; +} + +uptr SuspendedThreadsListMac::RegisterCount() const { + return MACHINE_THREAD_STATE_COUNT; +} +} // namespace __sanitizer + +#endif // SANITIZER_MAC && (defined(__x86_64__) || defined(__aarch64__)) || + // defined(__i386)) diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc index 7c377a7..614470a 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc @@ -356,11 +356,19 @@ const char *LLVMSymbolizer::FormatAndSendCommand(bool is_data, CHECK(module_name); const char *is_data_str = is_data ? "DATA " : ""; if (arch == kModuleArchUnknown) { - internal_snprintf(buffer_, kBufferSize, "%s\"%s\" 0x%zx\n", is_data_str, - module_name, module_offset); + if (internal_snprintf(buffer_, kBufferSize, "%s\"%s\" 0x%zx\n", is_data_str, + module_name, + module_offset) >= static_cast<int>(kBufferSize)) { + Report("WARNING: Command buffer too small"); + return nullptr; + } } else { - internal_snprintf(buffer_, kBufferSize, "%s\"%s:%s\" 0x%zx\n", is_data_str, - module_name, ModuleArchToString(arch), module_offset); + if (internal_snprintf(buffer_, kBufferSize, "%s\"%s:%s\" 0x%zx\n", + is_data_str, module_name, ModuleArchToString(arch), + module_offset) >= static_cast<int>(kBufferSize)) { + Report("WARNING: Command buffer too small"); + return nullptr; + } } return symbolizer_process_->SendCommand(buffer_); } @@ -377,7 +385,23 @@ SymbolizerProcess::SymbolizerProcess(const char *path, bool use_forkpty) CHECK_NE(path_[0], '\0'); } +static bool IsSameModule(const char* path) { + if (const char* ProcessName = GetProcessName()) { + if (const char* SymbolizerName = StripModuleName(path)) { + return !internal_strcmp(ProcessName, SymbolizerName); + } + } + return false; +} + const char *SymbolizerProcess::SendCommand(const char *command) { + if (failed_to_start_) + return nullptr; + if (IsSameModule(path_)) { + Report("WARNING: Symbolizer was blocked from starting itself!\n"); + failed_to_start_ = true; + return nullptr; + } for (; times_restarted_ < kMaxTimesRestarted; times_restarted_++) { // Start or restart symbolizer if we failed to send command to it. if (const char *res = SendCommandImpl(command)) @@ -426,6 +450,11 @@ bool SymbolizerProcess::ReadFromSymbolizer(char *buffer, uptr max_length) { read_len += just_read; if (ReachedEndOfOutput(buffer, read_len)) break; + if (read_len + 1 == max_length) { + Report("WARNING: Symbolizer buffer too small"); + read_len = 0; + break; + } } buffer[read_len] = '\0'; return true; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc index f50d8b1..60ec750 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc @@ -424,7 +424,6 @@ class InternalSymbolizer : public SymbolizerTool { InternalSymbolizer() { } static const int kBufferSize = 16 * 1024; - static const int kMaxDemangledNameSize = 1024; char buffer_[kBufferSize]; }; #else // SANITIZER_SUPPORTS_WEAK_HOOKS @@ -496,7 +495,7 @@ static void ChooseSymbolizerTools(IntrusiveList<SymbolizerTool> *list, VReport(2, "Symbolizer is disabled.\n"); return; } - if (IsReportingOOM()) { + if (IsAllocatorOutOfMemory()) { VReport(2, "Cannot use internal symbolizer: out of memory\n"); } else if (SymbolizerTool *tool = InternalSymbolizer::get(allocator)) { VReport(2, "Using internal symbolizer.\n"); diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.cc index c2b75e6..439e33a 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.cc @@ -19,7 +19,7 @@ namespace __sanitizer { ThreadContextBase::ThreadContextBase(u32 tid) : tid(tid), unique_id(0), reuse_count(), os_id(0), user_id(0), status(ThreadStatusInvalid), - detached(false), parent_tid(0), next(0) { + detached(false), workerthread(false), parent_tid(0), next(0) { name[0] = '\0'; } @@ -59,9 +59,11 @@ void ThreadContextBase::SetFinished() { OnFinished(); } -void ThreadContextBase::SetStarted(uptr _os_id, void *arg) { +void ThreadContextBase::SetStarted(tid_t _os_id, bool _workerthread, + void *arg) { status = ThreadStatusRunning; os_id = _os_id; + workerthread = _workerthread; OnStarted(arg); } @@ -192,7 +194,7 @@ static bool FindThreadContextByOsIdCallback(ThreadContextBase *tctx, tctx->status != ThreadStatusDead); } -ThreadContextBase *ThreadRegistry::FindThreadContextByOsIDLocked(uptr os_id) { +ThreadContextBase *ThreadRegistry::FindThreadContextByOsIDLocked(tid_t os_id) { return FindThreadContextLocked(FindThreadContextByOsIdCallback, (void *)os_id); } @@ -266,14 +268,15 @@ void ThreadRegistry::FinishThread(u32 tid) { } } -void ThreadRegistry::StartThread(u32 tid, uptr os_id, void *arg) { +void ThreadRegistry::StartThread(u32 tid, tid_t os_id, bool workerthread, + void *arg) { BlockingMutexLock l(&mtx_); running_threads_++; CHECK_LT(tid, n_contexts_); ThreadContextBase *tctx = threads_[tid]; CHECK_NE(tctx, 0); CHECK_EQ(ThreadStatusCreated, tctx->status); - tctx->SetStarted(os_id, arg); + tctx->SetStarted(os_id, workerthread, arg); } void ThreadRegistry::QuarantinePush(ThreadContextBase *tctx) { diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.h index a27bbb3..9aae875 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_thread_registry.h @@ -39,12 +39,13 @@ class ThreadContextBase { const u32 tid; // Thread ID. Main thread should have tid = 0. u64 unique_id; // Unique thread ID. u32 reuse_count; // Number of times this tid was reused. - uptr os_id; // PID (used for reporting). + tid_t os_id; // PID (used for reporting). uptr user_id; // Some opaque user thread id (e.g. pthread_t). char name[64]; // As annotated by user. ThreadStatus status; bool detached; + bool workerthread; u32 parent_tid; ThreadContextBase *next; // For storing thread contexts in a list. @@ -54,7 +55,7 @@ class ThreadContextBase { void SetDead(); void SetJoined(void *arg); void SetFinished(); - void SetStarted(uptr _os_id, void *arg); + void SetStarted(tid_t _os_id, bool _workerthread, void *arg); void SetCreated(uptr _user_id, u64 _unique_id, bool _detached, u32 _parent_tid, void *arg); void Reset(); @@ -108,14 +109,14 @@ class ThreadRegistry { // is found. ThreadContextBase *FindThreadContextLocked(FindThreadCallback cb, void *arg); - ThreadContextBase *FindThreadContextByOsIDLocked(uptr os_id); + ThreadContextBase *FindThreadContextByOsIDLocked(tid_t os_id); void SetThreadName(u32 tid, const char *name); void SetThreadNameByUserId(uptr user_id, const char *name); void DetachThread(u32 tid, void *arg); void JoinThread(u32 tid, void *arg); void FinishThread(u32 tid); - void StartThread(u32 tid, uptr os_id, void *arg); + void StartThread(u32 tid, tid_t os_id, bool workerthread, void *arg); private: const ThreadContextFactory context_factory_; diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.cc index 77c1947..29db37b 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.cc @@ -136,11 +136,19 @@ void DTLS_on_libc_memalign(void *ptr, uptr size) { DTLS *DTLS_Get() { return &dtls; } +bool DTLSInDestruction(DTLS *dtls) { + return dtls->dtv_size == kDestroyedThread; +} + #else void DTLS_on_libc_memalign(void *ptr, uptr size) {} DTLS::DTV *DTLS_on_tls_get_addr(void *arg, void *res) { return 0; } DTLS *DTLS_Get() { return 0; } void DTLS_Destroy() {} +bool DTLSInDestruction(DTLS *dtls) { + UNREACHABLE("dtls is unsupported on this platform!"); +} + #endif // SANITIZER_INTERCEPT_TLS_GET_ADDR } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.h index 58d4763..199a3b2 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.h +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_tls_get_addr.h @@ -55,6 +55,8 @@ DTLS::DTV *DTLS_on_tls_get_addr(void *arg, void *res, uptr static_tls_begin, void DTLS_on_libc_memalign(void *ptr, uptr size); DTLS *DTLS_Get(); void DTLS_Destroy(); // Make sure to call this before the thread is destroyed. +// Returns true if DTLS of suspended thread is in destruction process. +bool DTLSInDestruction(DTLS *dtls); } // namespace __sanitizer diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.cc index 9682d29..de01e8d 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.cc @@ -30,6 +30,7 @@ #include "sanitizer_procmaps.h" #include "sanitizer_stacktrace.h" #include "sanitizer_symbolizer.h" +#include "sanitizer_win_defs.h" // A macro to tell the compiler that this part of the code cannot be reached, // if the compiler supports this feature. Since we're using this in @@ -79,7 +80,7 @@ uptr internal_getpid() { // In contrast to POSIX, on Windows GetCurrentThreadId() // returns a system-unique identifier. -uptr GetTid() { +tid_t GetTid() { return GetCurrentThreadId(); } @@ -130,8 +131,24 @@ void UnmapOrDie(void *addr, uptr size) { } } +static void *ReturnNullptrOnOOMOrDie(uptr size, const char *mem_type, + const char *mmap_type) { + error_t last_error = GetLastError(); + if (last_error == ERROR_NOT_ENOUGH_MEMORY) + return nullptr; + ReportMmapFailureAndDie(size, mem_type, mmap_type, last_error); +} + +void *MmapOrDieOnFatalError(uptr size, const char *mem_type) { + void *rv = VirtualAlloc(0, size, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); + if (rv == 0) + return ReturnNullptrOnOOMOrDie(size, mem_type, "allocate"); + return rv; +} + // We want to map a chunk of address space aligned to 'alignment'. -void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type) { +void *MmapAlignedOrDieOnFatalError(uptr size, uptr alignment, + const char *mem_type) { CHECK(IsPowerOfTwo(size)); CHECK(IsPowerOfTwo(alignment)); @@ -141,7 +158,7 @@ void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type) { uptr mapped_addr = (uptr)VirtualAlloc(0, size, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); if (!mapped_addr) - ReportMmapFailureAndDie(size, mem_type, "allocate aligned", GetLastError()); + return ReturnNullptrOnOOMOrDie(size, mem_type, "allocate aligned"); // If we got it right on the first try, return. Otherwise, unmap it and go to // the slow path. @@ -161,8 +178,7 @@ void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type) { mapped_addr = (uptr)VirtualAlloc(0, size + alignment, MEM_RESERVE, PAGE_NOACCESS); if (!mapped_addr) - ReportMmapFailureAndDie(size, mem_type, "allocate aligned", - GetLastError()); + return ReturnNullptrOnOOMOrDie(size, mem_type, "allocate aligned"); // Find the aligned address. uptr aligned_addr = RoundUpTo(mapped_addr, alignment); @@ -180,7 +196,7 @@ void *MmapAlignedOrDie(uptr size, uptr alignment, const char *mem_type) { // Fail if we can't make this work quickly. if (retries == kMaxRetries && mapped_addr == 0) - ReportMmapFailureAndDie(size, mem_type, "allocate aligned", GetLastError()); + return ReturnNullptrOnOOMOrDie(size, mem_type, "allocate aligned"); return (void *)mapped_addr; } @@ -219,6 +235,18 @@ void *MmapFixedOrDie(uptr fixed_addr, uptr size) { return p; } +void *MmapFixedOrDieOnFatalError(uptr fixed_addr, uptr size) { + void *p = VirtualAlloc((LPVOID)fixed_addr, size, + MEM_COMMIT, PAGE_READWRITE); + if (p == 0) { + char mem_type[30]; + internal_snprintf(mem_type, sizeof(mem_type), "memory at address 0x%zx", + fixed_addr); + return ReturnNullptrOnOOMOrDie(size, mem_type, "allocate"); + } + return p; +} + void *MmapNoReserveOrDie(uptr size, const char *mem_type) { // FIXME: make this really NoReserve? return MmapOrDie(size, mem_type); @@ -263,7 +291,8 @@ void DontDumpShadowMemory(uptr addr, uptr length) { // FIXME: add madvise-analog when we move to 64-bits. } -uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding) { +uptr FindAvailableMemoryRange(uptr size, uptr alignment, uptr left_padding, + uptr *largest_gap_found) { uptr address = 0; while (true) { MEMORY_BASIC_INFORMATION info; @@ -399,9 +428,6 @@ void ReExec() { } void PrepareForSandboxing(__sanitizer_sandbox_arguments *args) { -#if !SANITIZER_GO - CovPrepareForSandboxing(args); -#endif } bool StackSizeIsUnlimited() { @@ -552,7 +578,8 @@ void ListOfModules::init() { LoadedModule cur_module; cur_module.set(module_name, adjusted_base); // We add the whole module as one single address range. - cur_module.addAddressRange(base_address, end_address, /*executable*/ true); + cur_module.addAddressRange(base_address, end_address, /*executable*/ true, + /*writable*/ true); modules_.push_back(cur_module); } UnmapOrDie(hmodules, modules_buffer_size); @@ -830,11 +857,64 @@ void InstallDeadlySignalHandlers(SignalHandlerType handler) { // FIXME: Decide what to do on Windows. } -bool IsHandledDeadlySignal(int signum) { +HandleSignalMode GetHandleSignalMode(int signum) { // FIXME: Decide what to do on Windows. + return kHandleSignalNo; +} + +// Check based on flags if we should handle this exception. +bool IsHandledDeadlyException(DWORD exceptionCode) { + switch (exceptionCode) { + case EXCEPTION_ACCESS_VIOLATION: + case EXCEPTION_ARRAY_BOUNDS_EXCEEDED: + case EXCEPTION_STACK_OVERFLOW: + case EXCEPTION_DATATYPE_MISALIGNMENT: + case EXCEPTION_IN_PAGE_ERROR: + return common_flags()->handle_segv; + case EXCEPTION_ILLEGAL_INSTRUCTION: + case EXCEPTION_PRIV_INSTRUCTION: + case EXCEPTION_BREAKPOINT: + return common_flags()->handle_sigill; + case EXCEPTION_FLT_DENORMAL_OPERAND: + case EXCEPTION_FLT_DIVIDE_BY_ZERO: + case EXCEPTION_FLT_INEXACT_RESULT: + case EXCEPTION_FLT_INVALID_OPERATION: + case EXCEPTION_FLT_OVERFLOW: + case EXCEPTION_FLT_STACK_CHECK: + case EXCEPTION_FLT_UNDERFLOW: + case EXCEPTION_INT_DIVIDE_BY_ZERO: + case EXCEPTION_INT_OVERFLOW: + return common_flags()->handle_sigfpe; + } return false; } +const char *DescribeSignalOrException(int signo) { + unsigned code = signo; + // Get the string description of the exception if this is a known deadly + // exception. + switch (code) { + case EXCEPTION_ACCESS_VIOLATION: return "access-violation"; + case EXCEPTION_ARRAY_BOUNDS_EXCEEDED: return "array-bounds-exceeded"; + case EXCEPTION_STACK_OVERFLOW: return "stack-overflow"; + case EXCEPTION_DATATYPE_MISALIGNMENT: return "datatype-misalignment"; + case EXCEPTION_IN_PAGE_ERROR: return "in-page-error"; + case EXCEPTION_ILLEGAL_INSTRUCTION: return "illegal-instruction"; + case EXCEPTION_PRIV_INSTRUCTION: return "priv-instruction"; + case EXCEPTION_BREAKPOINT: return "breakpoint"; + case EXCEPTION_FLT_DENORMAL_OPERAND: return "flt-denormal-operand"; + case EXCEPTION_FLT_DIVIDE_BY_ZERO: return "flt-divide-by-zero"; + case EXCEPTION_FLT_INEXACT_RESULT: return "flt-inexact-result"; + case EXCEPTION_FLT_INVALID_OPERATION: return "flt-invalid-operation"; + case EXCEPTION_FLT_OVERFLOW: return "flt-overflow"; + case EXCEPTION_FLT_STACK_CHECK: return "flt-stack-check"; + case EXCEPTION_FLT_UNDERFLOW: return "flt-underflow"; + case EXCEPTION_INT_DIVIDE_BY_ZERO: return "int-divide-by-zero"; + case EXCEPTION_INT_OVERFLOW: return "int-overflow"; + } + return "unknown exception"; +} + bool IsAccessibleMemoryRange(uptr beg, uptr size) { SYSTEM_INFO si; GetNativeSystemInfo(&si); @@ -936,21 +1016,15 @@ int WaitForProcess(pid_t pid) { return -1; } // FIXME implement on this platform. void GetMemoryProfile(fill_profile_f cb, uptr *stats, uptr stats_size) { } +void CheckNoDeepBind(const char *filename, int flag) { + // Do nothing. +} -} // namespace __sanitizer - -#if !SANITIZER_GO -// Workaround to implement weak hooks on Windows. COFF doesn't directly support -// weak symbols, but it does support /alternatename, which is similar. If the -// user does not override the hook, we will use this default definition instead -// of null. -extern "C" void __sanitizer_print_memory_profile(int top_percent) {} +// FIXME: implement on this platform. +bool GetRandom(void *buffer, uptr length) { + UNIMPLEMENTED(); +} -#ifdef _WIN64 -#pragma comment(linker, "/alternatename:__sanitizer_print_memory_profile=__sanitizer_default_print_memory_profile") // NOLINT -#else -#pragma comment(linker, "/alternatename:___sanitizer_print_memory_profile=___sanitizer_default_print_memory_profile") // NOLINT -#endif -#endif +} // namespace __sanitizer #endif // _WIN32 diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.h new file mode 100644 index 0000000..23e01ab --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win.h @@ -0,0 +1,26 @@ +//===-- sanitizer_win.h -----------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// Windows-specific declarations. +// +//===----------------------------------------------------------------------===// +#ifndef SANITIZER_WIN_H +#define SANITIZER_WIN_H + +#include "sanitizer_platform.h" +#if SANITIZER_WINDOWS +#include "sanitizer_internal_defs.h" + +namespace __sanitizer { +// Check based on flags if we should handle the exception. +bool IsHandledDeadlyException(DWORD exceptionCode); +} // namespace __sanitizer + +#endif // SANITIZER_WINDOWS +#endif // SANITIZER_WIN_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_defs.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_defs.h new file mode 100644 index 0000000..077ff9c --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_defs.h @@ -0,0 +1,153 @@ +//===-- sanitizer_win_defs.h ------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// Common definitions for Windows-specific code. +// +//===----------------------------------------------------------------------===// +#ifndef SANITIZER_WIN_DEFS_H +#define SANITIZER_WIN_DEFS_H + +#include "sanitizer_platform.h" +#if SANITIZER_WINDOWS + +#ifndef WINAPI +#ifdef _M_IX86 +#define WINAPI __stdcall +#else +#define WINAPI +#endif +#endif + +#if defined(_WIN64) +#define WIN_SYM_PREFIX +#else +#define WIN_SYM_PREFIX "_" +#endif + +// Intermediate macro to ensure the parameter is expanded before stringified. +#define STRINGIFY_(A) #A +#define STRINGIFY(A) STRINGIFY_(A) + +// ----------------- A workaround for the absence of weak symbols -------------- +// We don't have a direct equivalent of weak symbols when using MSVC, but we can +// use the /alternatename directive to tell the linker to default a specific +// symbol to a specific value. +// Take into account that this is a pragma directive for the linker, so it will +// be ignored by the compiler and the function will be marked as UNDEF in the +// symbol table of the resulting object file. The linker won't find the default +// implementation until it links with that object file. +// So, suppose we provide a default implementation "fundef" for "fun", and this +// is compiled into the object file "test.obj" including the pragma directive. +// If we have some code with references to "fun" and we link that code with +// "test.obj", it will work because the linker always link object files. +// But, if "test.obj" is included in a static library, like "test.lib", then the +// liker will only link to "test.obj" if necessary. If we only included the +// definition of "fun", it won't link to "test.obj" (from test.lib) because +// "fun" appears as UNDEF, so it doesn't resolve the symbol "fun", and will +// result in a link error (the linker doesn't find the pragma directive). +// So, a workaround is to force linkage with the modules that include weak +// definitions, with the following macro: WIN_FORCE_LINK() + +#define WIN_WEAK_ALIAS(Name, Default) \ + __pragma(comment(linker, "/alternatename:" WIN_SYM_PREFIX STRINGIFY(Name) "="\ + WIN_SYM_PREFIX STRINGIFY(Default))) + +#define WIN_FORCE_LINK(Name) \ + __pragma(comment(linker, "/include:" WIN_SYM_PREFIX STRINGIFY(Name))) + +#define WIN_EXPORT(ExportedName, Name) \ + __pragma(comment(linker, "/export:" WIN_SYM_PREFIX STRINGIFY(ExportedName) \ + "=" WIN_SYM_PREFIX STRINGIFY(Name))) + +// We cannot define weak functions on Windows, but we can use WIN_WEAK_ALIAS() +// which defines an alias to a default implementation, and only works when +// linking statically. +// So, to define a weak function "fun", we define a default implementation with +// a different name "fun__def" and we create a "weak alias" fun = fun__def. +// Then, users can override it just defining "fun". +// We impose "extern "C"" because otherwise WIN_WEAK_ALIAS() will fail because +// of name mangling. + +// Dummy name for default implementation of weak function. +# define WEAK_DEFAULT_NAME(Name) Name##__def +// Name for exported implementation of weak function. +# define WEAK_EXPORT_NAME(Name) Name##__dll + +// Use this macro when you need to define and export a weak function from a +// library. For example: +// WIN_WEAK_EXPORT_DEF(bool, compare, int a, int b) { return a > b; } +# define WIN_WEAK_EXPORT_DEF(ReturnType, Name, ...) \ + WIN_WEAK_ALIAS(Name, WEAK_DEFAULT_NAME(Name)) \ + WIN_EXPORT(WEAK_EXPORT_NAME(Name), Name) \ + extern "C" ReturnType Name(__VA_ARGS__); \ + extern "C" ReturnType WEAK_DEFAULT_NAME(Name)(__VA_ARGS__) + +// Use this macro when you need to import a weak function from a library. It +// defines a weak alias to the imported function from the dll. For example: +// WIN_WEAK_IMPORT_DEF(compare) +# define WIN_WEAK_IMPORT_DEF(Name) \ + WIN_WEAK_ALIAS(Name, WEAK_EXPORT_NAME(Name)) + +// So, for Windows we provide something similar to weak symbols in Linux, with +// some differences: +// + A default implementation must always be provided. +// +// + When linking statically it works quite similarly. For example: +// +// // libExample.cc +// WIN_WEAK_EXPORT_DEF(bool, compare, int a, int b) { return a > b; } +// +// // client.cc +// // We can use the default implementation from the library: +// compare(1, 2); +// // Or we can override it: +// extern "C" bool compare (int a, int b) { return a >= b; } +// +// And it will work fine. If we don't override the function, we need to ensure +// that the linker includes the object file with the default implementation. +// We can do so with the linker option "-wholearchive:". +// +// + When linking dynamically with a library (dll), weak functions are exported +// with "__dll" suffix. Clients can use the macro WIN_WEAK_IMPORT_DEF(fun) +// which defines a "weak alias" fun = fun__dll. +// +// // libExample.cc +// WIN_WEAK_EXPORT_DEF(bool, compare, int a, int b) { return a > b; } +// +// // client.cc +// WIN_WEAK_IMPORT_DEF(compare) +// // We can use the default implementation from the library: +// compare(1, 2); +// // Or we can override it: +// extern "C" bool compare (int a, int b) { return a >= b; } +// +// But if we override the function, the dlls don't have access to it (which +// is different in linux). If that is desired, the strong definition must be +// exported and interception can be used from the rest of the dlls. +// +// // libExample.cc +// WIN_WEAK_EXPORT_DEF(bool, compare, int a, int b) { return a > b; } +// // When initialized, check if the main executable defined "compare". +// int libExample_init() { +// uptr fnptr = __interception::InternalGetProcAddress( +// (void *)GetModuleHandleA(0), "compare"); +// if (fnptr && !__interception::OverrideFunction((uptr)compare, fnptr, 0)) +// abort(); +// return 0; +// } +// +// // client.cc +// WIN_WEAK_IMPORT_DEF(compare) +// // We override and export compare: +// extern "C" __declspec(dllexport) bool compare (int a, int b) { +// return a >= b; +// } +// +#endif // SANITIZER_WINDOWS +#endif // SANITIZER_WIN_DEFS_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.cc new file mode 100644 index 0000000..4fb4650 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.cc @@ -0,0 +1,102 @@ +//===-- sanitizer_win_dll_thunk.cc ----------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This file defines a family of thunks that should be statically linked into +// the DLLs that have instrumentation in order to delegate the calls to the +// shared runtime that lives in the main binary. +// See https://github.com/google/sanitizers/issues/209 for the details. +//===----------------------------------------------------------------------===// + +#ifdef SANITIZER_DLL_THUNK +#include "sanitizer_win_defs.h" +#include "sanitizer_win_dll_thunk.h" +#include "interception/interception.h" + +extern "C" { +void *WINAPI GetModuleHandleA(const char *module_name); +void abort(); +} + +namespace __sanitizer { +uptr dllThunkGetRealAddrOrDie(const char *name) { + uptr ret = + __interception::InternalGetProcAddress((void *)GetModuleHandleA(0), name); + if (!ret) + abort(); + return ret; +} + +int dllThunkIntercept(const char* main_function, uptr dll_function) { + uptr wrapper = dllThunkGetRealAddrOrDie(main_function); + if (!__interception::OverrideFunction(dll_function, wrapper, 0)) + abort(); + return 0; +} + +int dllThunkInterceptWhenPossible(const char* main_function, + const char* default_function, uptr dll_function) { + uptr wrapper = __interception::InternalGetProcAddress( + (void *)GetModuleHandleA(0), main_function); + if (!wrapper) + wrapper = dllThunkGetRealAddrOrDie(default_function); + if (!__interception::OverrideFunction(dll_function, wrapper, 0)) + abort(); + return 0; +} +} // namespace __sanitizer + +// Include Sanitizer Common interface. +#define INTERFACE_FUNCTION(Name) INTERCEPT_SANITIZER_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "sanitizer_common_interface.inc" + +#pragma section(".DLLTH$A", read) // NOLINT +#pragma section(".DLLTH$Z", read) // NOLINT + +typedef void (*DllThunkCB)(); +extern "C" { +__declspec(allocate(".DLLTH$A")) DllThunkCB __start_dll_thunk; +__declspec(allocate(".DLLTH$Z")) DllThunkCB __stop_dll_thunk; +} + +// Disable compiler warnings that show up if we declare our own version +// of a compiler intrinsic (e.g. strlen). +#pragma warning(disable: 4391) +#pragma warning(disable: 4392) + +extern "C" int __dll_thunk_init() { + static bool flag = false; + // __dll_thunk_init is expected to be called by only one thread. + if (flag) return 0; + flag = true; + + for (DllThunkCB *it = &__start_dll_thunk; it < &__stop_dll_thunk; ++it) + if (*it) + (*it)(); + + // In DLLs, the callbacks are expected to return 0, + // otherwise CRT initialization fails. + return 0; +} + +// We want to call dll_thunk_init before C/C++ initializers / constructors are +// executed, otherwise functions like memset might be invoked. +#pragma section(".CRT$XIB", long, read) // NOLINT +__declspec(allocate(".CRT$XIB")) int (*__dll_thunk_preinit)() = + __dll_thunk_init; + +static void WINAPI dll_thunk_thread_init(void *mod, unsigned long reason, + void *reserved) { + if (reason == /*DLL_PROCESS_ATTACH=*/1) __dll_thunk_init(); +} + +#pragma section(".CRT$XLAB", long, read) // NOLINT +__declspec(allocate(".CRT$XLAB")) void (WINAPI *__dll_thunk_tls_init)(void *, + unsigned long, void *) = dll_thunk_thread_init; + +#endif // SANITIZER_DLL_THUNK diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.h new file mode 100644 index 0000000..2f9ebda --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dll_thunk.h @@ -0,0 +1,182 @@ +//===-- sanitizer_win_dll_thunk.h -----------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This header provide helper macros to delegate calls to the shared runtime +// that lives in the main executable. It should be included to dll_thunks that +// will be linked to the dlls, when the sanitizer is a static library included +// in the main executable. +//===----------------------------------------------------------------------===// +#ifndef SANITIZER_WIN_DLL_THUNK_H +#define SANITIZER_WIN_DLL_THUNK_H +#include "sanitizer_internal_defs.h" + +namespace __sanitizer { +uptr dllThunkGetRealAddrOrDie(const char *name); + +int dllThunkIntercept(const char* main_function, uptr dll_function); + +int dllThunkInterceptWhenPossible(const char* main_function, + const char* default_function, uptr dll_function); +} + +extern "C" int __dll_thunk_init(); + +// ----------------- Function interception helper macros -------------------- // +// Override dll_function with main_function from main executable. +#define INTERCEPT_OR_DIE(main_function, dll_function) \ + static int intercept_##dll_function() { \ + return __sanitizer::dllThunkIntercept(main_function, (__sanitizer::uptr) \ + dll_function); \ + } \ + __pragma(section(".DLLTH$M", long, read)) \ + __declspec(allocate(".DLLTH$M")) int (*__dll_thunk_##dll_function)() = \ + intercept_##dll_function; + +// Try to override dll_function with main_function from main executable. +// If main_function is not present, override dll_function with default_function. +#define INTERCEPT_WHEN_POSSIBLE(main_function, default_function, dll_function) \ + static int intercept_##dll_function() { \ + return __sanitizer::dllThunkInterceptWhenPossible(main_function, \ + default_function, (__sanitizer::uptr)dll_function); \ + } \ + __pragma(section(".DLLTH$M", long, read)) \ + __declspec(allocate(".DLLTH$M")) int (*__dll_thunk_##dll_function)() = \ + intercept_##dll_function; + +// -------------------- Function interception macros ------------------------ // +// Special case of hooks -- ASan own interface functions. Those are only called +// after __asan_init, thus an empty implementation is sufficient. +#define INTERCEPT_SANITIZER_FUNCTION(name) \ + extern "C" __declspec(noinline) void name() { \ + volatile int prevent_icf = (__LINE__ << 8) ^ __COUNTER__; \ + static const char function_name[] = #name; \ + for (const char* ptr = &function_name[0]; *ptr; ++ptr) \ + prevent_icf ^= *ptr; \ + (void)prevent_icf; \ + __debugbreak(); \ + } \ + INTERCEPT_OR_DIE(#name, name) + +// Special case of hooks -- Weak functions, could be redefined in the main +// executable, but that is not necessary, so we shouldn't die if we can not find +// a reference. Instead, when the function is not present in the main executable +// we consider the default impl provided by asan library. +#define INTERCEPT_SANITIZER_WEAK_FUNCTION(name) \ + extern "C" __declspec(noinline) void name() { \ + volatile int prevent_icf = (__LINE__ << 8) ^ __COUNTER__; \ + static const char function_name[] = #name; \ + for (const char* ptr = &function_name[0]; *ptr; ++ptr) \ + prevent_icf ^= *ptr; \ + (void)prevent_icf; \ + __debugbreak(); \ + } \ + INTERCEPT_WHEN_POSSIBLE(#name, STRINGIFY(WEAK_EXPORT_NAME(name)), name) + +// We can't define our own version of strlen etc. because that would lead to +// link-time or even type mismatch errors. Instead, we can declare a function +// just to be able to get its address. Me may miss the first few calls to the +// functions since it can be called before __dll_thunk_init, but that would lead +// to false negatives in the startup code before user's global initializers, +// which isn't a big deal. +#define INTERCEPT_LIBRARY_FUNCTION(name) \ + extern "C" void name(); \ + INTERCEPT_OR_DIE(WRAPPER_NAME(name), name) + +// Use these macros for functions that could be called before __dll_thunk_init() +// is executed and don't lead to errors if defined (free, malloc, etc). +#define INTERCEPT_WRAP_V_V(name) \ + extern "C" void name() { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + fn(); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_V_W(name) \ + extern "C" void name(void *arg) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + fn(arg); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_V_WW(name) \ + extern "C" void name(void *arg1, void *arg2) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + fn(arg1, arg2); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_V_WWW(name) \ + extern "C" void name(void *arg1, void *arg2, void *arg3) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + fn(arg1, arg2, arg3); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_V(name) \ + extern "C" void *name() { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_W(name) \ + extern "C" void *name(void *arg) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_WW(name) \ + extern "C" void *name(void *arg1, void *arg2) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg1, arg2); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_WWW(name) \ + extern "C" void *name(void *arg1, void *arg2, void *arg3) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg1, arg2, arg3); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_WWWW(name) \ + extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg1, arg2, arg3, arg4); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_WWWWW(name) \ + extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4, \ + void *arg5) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg1, arg2, arg3, arg4, arg5); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#define INTERCEPT_WRAP_W_WWWWWW(name) \ + extern "C" void *name(void *arg1, void *arg2, void *arg3, void *arg4, \ + void *arg5, void *arg6) { \ + typedef decltype(name) *fntype; \ + static fntype fn = (fntype)__sanitizer::dllThunkGetRealAddrOrDie(#name); \ + return fn(arg1, arg2, arg3, arg4, arg5, arg6); \ + } \ + INTERCEPT_OR_DIE(#name, name); + +#endif // SANITIZER_WIN_DLL_THUNK_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dynamic_runtime_thunk.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dynamic_runtime_thunk.cc new file mode 100644 index 0000000..f8f91647 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_dynamic_runtime_thunk.cc @@ -0,0 +1,21 @@ +//===-- santizer_win_dynamic_runtime_thunk.cc -----------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines things that need to be present in the application modules +// to interact with Sanitizer Common, when it is included in a dll. +// +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC_RUNTIME_THUNK +#define SANITIZER_IMPORT_INTERFACE 1 +#include "sanitizer_win_defs.h" +// Define weak alias for all weak functions imported from sanitizer common. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) WIN_WEAK_IMPORT_DEF(Name) +#include "sanitizer_common_interface.inc" +#endif // SANITIZER_DYNAMIC_RUNTIME_THUNK diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.cc b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.cc new file mode 100644 index 0000000..3643193 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.cc @@ -0,0 +1,94 @@ +//===-- sanitizer_win_weak_interception.cc --------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This module should be included in the sanitizer when it is implemented as a +// shared library on Windows (dll), in order to delegate the calls of weak +// functions to the implementation in the main executable when a strong +// definition is provided. +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" +#if SANITIZER_WINDOWS && SANITIZER_DYNAMIC +#include "sanitizer_win_weak_interception.h" +#include "sanitizer_allocator_interface.h" +#include "sanitizer_interface_internal.h" +#include "sanitizer_win_defs.h" +#include "interception/interception.h" + +extern "C" { +void *WINAPI GetModuleHandleA(const char *module_name); +void abort(); +} + +namespace __sanitizer { +// Try to get a pointer to real_function in the main module and override +// dll_function with that pointer. If the function isn't found, nothing changes. +int interceptWhenPossible(uptr dll_function, const char *real_function) { + uptr real = __interception::InternalGetProcAddress( + (void *)GetModuleHandleA(0), real_function); + if (real && !__interception::OverrideFunction((uptr)dll_function, real, 0)) + abort(); + return 0; +} +} // namespace __sanitizer + +// Declare weak hooks. +extern "C" { +void __sanitizer_weak_hook_memcmp(uptr called_pc, const void *s1, + const void *s2, uptr n, int result); +void __sanitizer_weak_hook_strcmp(uptr called_pc, const char *s1, + const char *s2, int result); +void __sanitizer_weak_hook_strncmp(uptr called_pc, const char *s1, + const char *s2, uptr n, int result); +void __sanitizer_weak_hook_strstr(uptr called_pc, const char *s1, + const char *s2, char *result); +} + +// Include Sanitizer Common interface. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "sanitizer_common_interface.inc" + +#pragma section(".WEAK$A", read) // NOLINT +#pragma section(".WEAK$Z", read) // NOLINT + +typedef void (*InterceptCB)(); +extern "C" { +__declspec(allocate(".WEAK$A")) InterceptCB __start_weak_list; +__declspec(allocate(".WEAK$Z")) InterceptCB __stop_weak_list; +} + +static int weak_intercept_init() { + static bool flag = false; + // weak_interception_init is expected to be called by only one thread. + if (flag) return 0; + flag = true; + + for (InterceptCB *it = &__start_weak_list; it < &__stop_weak_list; ++it) + if (*it) + (*it)(); + + // In DLLs, the callbacks are expected to return 0, + // otherwise CRT initialization fails. + return 0; +} + +#pragma section(".CRT$XIB", long, read) // NOLINT +__declspec(allocate(".CRT$XIB")) int (*__weak_intercept_preinit)() = + weak_intercept_init; + +static void WINAPI weak_intercept_thread_init(void *mod, unsigned long reason, + void *reserved) { + if (reason == /*DLL_PROCESS_ATTACH=*/1) weak_intercept_init(); +} + +#pragma section(".CRT$XLAB", long, read) // NOLINT +__declspec(allocate(".CRT$XLAB")) void(WINAPI *__weak_intercept_tls_init)( + void *, unsigned long, void *) = weak_intercept_thread_init; + +#endif // SANITIZER_WINDOWS && SANITIZER_DYNAMIC diff --git a/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.h b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.h new file mode 100644 index 0000000..5b12297 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/sanitizer_win_weak_interception.h @@ -0,0 +1,33 @@ +//===-- sanitizer_win_weak_interception.h ---------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This header provide helper macros to delegate calls of weak functions to the +// implementation in the main executable when a strong definition is present. +//===----------------------------------------------------------------------===// +#ifndef SANITIZER_WIN_WEAK_INTERCEPTION_H +#define SANITIZER_WIN_WEAK_INTERCEPTION_H +#include "sanitizer_internal_defs.h" + +namespace __sanitizer { +int interceptWhenPossible(uptr dll_function, const char *real_function); +} + +// ----------------- Function interception helper macros -------------------- // +// Weak functions, could be redefined in the main executable, but that is not +// necessary, so we shouldn't die if we can not find a reference. +#define INTERCEPT_WEAK(Name) interceptWhenPossible((uptr) Name, #Name); + +#define INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) \ + static int intercept_##Name() { \ + return __sanitizer::interceptWhenPossible((__sanitizer::uptr) Name, #Name);\ + } \ + __pragma(section(".WEAK$M", long, read)) \ + __declspec(allocate(".WEAK$M")) int (*__weak_intercept_##Name)() = \ + intercept_##Name; + +#endif // SANITIZER_WIN_WEAK_INTERCEPTION_H diff --git a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_symbolize.cc b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_symbolize.cc index bd315a0..b25a53d 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_symbolize.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_symbolize.cc @@ -18,8 +18,9 @@ #include "llvm/DebugInfo/Symbolize/Symbolize.h" static llvm::symbolize::LLVMSymbolizer *getDefaultSymbolizer() { - static llvm::symbolize::LLVMSymbolizer DefaultSymbolizer; - return &DefaultSymbolizer; + static llvm::symbolize::LLVMSymbolizer *DefaultSymbolizer = + new llvm::symbolize::LLVMSymbolizer(); + return DefaultSymbolizer; } namespace __sanitizer { @@ -41,8 +42,8 @@ bool __sanitizer_symbolize_code(const char *ModuleName, uint64_t ModuleOffset, getDefaultSymbolizer()->symbolizeInlinedCode(ModuleName, ModuleOffset); Printer << (ResOrErr ? ResOrErr.get() : llvm::DIInliningInfo()); } - __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", Result.c_str()); - return true; + return __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", + Result.c_str()) < MaxLength; } bool __sanitizer_symbolize_data(const char *ModuleName, uint64_t ModuleOffset, @@ -55,8 +56,8 @@ bool __sanitizer_symbolize_data(const char *ModuleName, uint64_t ModuleOffset, getDefaultSymbolizer()->symbolizeData(ModuleName, ModuleOffset); Printer << (ResOrErr ? ResOrErr.get() : llvm::DIGlobal()); } - __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", Result.c_str()); - return true; + return __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", + Result.c_str()) < MaxLength; } void __sanitizer_symbolize_flush() { getDefaultSymbolizer()->flush(); } @@ -65,8 +66,10 @@ int __sanitizer_symbolize_demangle(const char *Name, char *Buffer, int MaxLength) { std::string Result = llvm::symbolize::LLVMSymbolizer::DemangleName(Name, nullptr); - __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", Result.c_str()); - return static_cast<int>(Result.size() + 1); + return __sanitizer::internal_snprintf(Buffer, MaxLength, "%s", + Result.c_str()) < MaxLength + ? static_cast<int>(Result.size() + 1) + : 0; } } // extern "C" diff --git a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_wrappers.cc b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_wrappers.cc index 0a796d9..66d089a 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_wrappers.cc +++ b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/sanitizer_wrappers.cc @@ -172,4 +172,28 @@ LLVM_SYMBOLIZER_INTERCEPTOR4(pread, ssize_t(int, void *, size_t, off_t)) LLVM_SYMBOLIZER_INTERCEPTOR4(pread64, ssize_t(int, void *, size_t, off64_t)) LLVM_SYMBOLIZER_INTERCEPTOR2(realpath, char *(const char *, char *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_cond_broadcast, int(pthread_cond_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_cond_wait, + int(pthread_cond_t *, pthread_mutex_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_mutex_lock, int(pthread_mutex_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_mutex_unlock, int(pthread_mutex_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_mutex_destroy, int(pthread_mutex_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_mutex_init, + int(pthread_mutex_t *, + const pthread_mutexattr_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_mutexattr_destroy, + int(pthread_mutexattr_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_mutexattr_init, int(pthread_mutexattr_t *)) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_mutexattr_settype, + int(pthread_mutexattr_t *, int)) +LLVM_SYMBOLIZER_INTERCEPTOR1(pthread_getspecific, void *(pthread_key_t)) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_key_create, + int(pthread_key_t *, void (*)(void *))) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_once, + int(pthread_once_t *, void (*)(void))) +LLVM_SYMBOLIZER_INTERCEPTOR2(pthread_setspecific, + int(pthread_key_t, const void *)) +LLVM_SYMBOLIZER_INTERCEPTOR3(pthread_sigmask, + int(int, const sigset_t *, sigset_t *)) + } // extern "C" diff --git a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/ar_to_bc.sh b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/ar_to_bc.sh new file mode 100755 index 0000000..788cef8 --- /dev/null +++ b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/ar_to_bc.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +function usage() { + echo "Usage: $0 INPUT... OUTPUT" + exit 1 +} + +if [ "$#" -le 1 ]; then + usage +fi + +AR=$(readlink -f $AR) +LINK=$(readlink -f $LINK) + +INPUTS= +OUTPUT= +for ARG in $@; do + INPUTS="$INPUTS $OUTPUT" + OUTPUT=$(readlink -f $ARG) +done + +echo Inputs: $INPUTS +echo Output: $OUTPUT + +SCRATCH_DIR=$(mktemp -d) +ln -s $INPUTS $SCRATCH_DIR/ + +pushd $SCRATCH_DIR + +for INPUT in *; do + for OBJ in $($AR t $INPUT); do + $AR x $INPUT $OBJ + mv -f $OBJ $(basename $INPUT).$OBJ + done +done + +$LINK *.o -o $OUTPUT + +rm -rf $SCRATCH_DIR diff --git a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh index 07239eb..c5865ec 100755 --- a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh +++ b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh @@ -58,9 +58,9 @@ cd $BUILD_DIR CC=$CLANG_DIR/clang CXX=$CLANG_DIR/clang++ TBLGEN=$CLANG_DIR/llvm-tblgen -LINK=$CLANG_DIR/llvm-link OPT=$CLANG_DIR/opt -AR=$CLANG_DIR/llvm-ar +export AR=$CLANG_DIR/llvm-ar +export LINK=$CLANG_DIR/llvm-link for F in $CC $CXX $TBLGEN $LINK $OPT $AR; do if [[ ! -x "$F" ]]; then @@ -129,36 +129,34 @@ if [[ ! -d ${LLVM_BUILD} ]]; then $LLVM_SRC fi cd ${LLVM_BUILD} -ninja LLVMSymbolize LLVMObject LLVMDebugInfoDWARF LLVMSupport LLVMDebugInfoPDB LLVMMC +ninja LLVMSymbolize LLVMObject LLVMBinaryFormat LLVMDebugInfoDWARF LLVMSupport LLVMDebugInfoPDB LLVMMC cd ${BUILD_DIR} rm -rf ${SYMBOLIZER_BUILD} mkdir ${SYMBOLIZER_BUILD} cd ${SYMBOLIZER_BUILD} -for A in $LIBCXX_BUILD/lib/libc++.a \ - $LIBCXX_BUILD/lib/libc++abi.a \ - $LLVM_BUILD/lib/libLLVMSymbolize.a \ - $LLVM_BUILD/lib/libLLVMObject.a \ - $LLVM_BUILD/lib/libLLVMDebugInfoDWARF.a \ - $LLVM_BUILD/lib/libLLVMSupport.a \ - $LLVM_BUILD/lib/libLLVMDebugInfoPDB.a \ - $LLVM_BUILD/lib/libLLVMMC.a \ - $ZLIB_BUILD/libz.a ; do - for O in $($AR t $A); do - $AR x $A $O - mv -f $O "$(basename $A).$O" # Rename to avoid collisions between libs. - done -done - echo "Compiling..." SYMBOLIZER_FLAGS="$FLAGS -std=c++11 -I${LLVM_SRC}/include -I${LLVM_BUILD}/include -I${LIBCXX_BUILD}/include/c++/v1" $CXX $SYMBOLIZER_FLAGS ${SRC_DIR}/sanitizer_symbolize.cc ${SRC_DIR}/sanitizer_wrappers.cc -c +$AR rc symbolizer.a sanitizer_symbolize.o sanitizer_wrappers.o SYMBOLIZER_API_LIST=__sanitizer_symbolize_code,__sanitizer_symbolize_data,__sanitizer_symbolize_flush,__sanitizer_symbolize_demangle # Merge all the object files together and copy the resulting library back. -$LINK *.o -o all.bc +$SCRIPT_DIR/ar_to_bc.sh $LIBCXX_BUILD/lib/libc++.a \ + $LIBCXX_BUILD/lib/libc++abi.a \ + $LLVM_BUILD/lib/libLLVMSymbolize.a \ + $LLVM_BUILD/lib/libLLVMObject.a \ + $LLVM_BUILD/lib/libLLVMBinaryFormat.a \ + $LLVM_BUILD/lib/libLLVMDebugInfoDWARF.a \ + $LLVM_BUILD/lib/libLLVMSupport.a \ + $LLVM_BUILD/lib/libLLVMDebugInfoPDB.a \ + $LLVM_BUILD/lib/libLLVMMC.a \ + $ZLIB_BUILD/libz.a \ + symbolizer.a \ + all.bc + echo "Optimizing..." $OPT -internalize -internalize-public-api-list=${SYMBOLIZER_API_LIST} all.bc -o opt.bc $CC $FLAGS -fno-lto -c opt.bc -o symbolizer.o diff --git a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/global_symbols.txt b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/global_symbols.txt index 033acf7..a23c953 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/global_symbols.txt +++ b/contrib/compiler-rt/lib/sanitizer_common/symbolizer/scripts/global_symbols.txt @@ -37,6 +37,7 @@ clock_gettime U cfgetospeed U dl_iterate_phdr U dlsym U +dup U dup2 U environ U execv U @@ -58,6 +59,7 @@ getpagesize U getpid U gettimeofday U ioctl U +isalpha U isatty U isprint U isupper U diff --git a/contrib/compiler-rt/lib/sanitizer_common/weak_symbols.txt b/contrib/compiler-rt/lib/sanitizer_common/weak_symbols.txt index 8a1e32b..5a2b275 100644 --- a/contrib/compiler-rt/lib/sanitizer_common/weak_symbols.txt +++ b/contrib/compiler-rt/lib/sanitizer_common/weak_symbols.txt @@ -1,5 +1,7 @@ ___sanitizer_free_hook ___sanitizer_malloc_hook +___sanitizer_report_error_summary +___sanitizer_sandbox_on_notify ___sanitizer_symbolize_code ___sanitizer_symbolize_data ___sanitizer_symbolize_demangle diff --git a/contrib/compiler-rt/lib/scudo/scudo_allocator.cpp b/contrib/compiler-rt/lib/scudo/scudo_allocator.cpp index d1121b0..6f30ee9 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_allocator.cpp +++ b/contrib/compiler-rt/lib/scudo/scudo_allocator.cpp @@ -16,56 +16,18 @@ #include "scudo_allocator.h" #include "scudo_crc32.h" +#include "scudo_tls.h" #include "scudo_utils.h" +#include "sanitizer_common/sanitizer_allocator_checks.h" #include "sanitizer_common/sanitizer_allocator_interface.h" +#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_quarantine.h" -#include <limits.h> -#include <pthread.h> - -#include <cstring> +#include <string.h> namespace __scudo { -#if SANITIZER_CAN_USE_ALLOCATOR64 -const uptr AllocatorSpace = ~0ULL; -const uptr AllocatorSize = 0x40000000000ULL; -typedef DefaultSizeClassMap SizeClassMap; -struct AP { - static const uptr kSpaceBeg = AllocatorSpace; - static const uptr kSpaceSize = AllocatorSize; - static const uptr kMetadataSize = 0; - typedef __scudo::SizeClassMap SizeClassMap; - typedef NoOpMapUnmapCallback MapUnmapCallback; - static const uptr kFlags = - SizeClassAllocator64FlagMasks::kRandomShuffleChunks; -}; -typedef SizeClassAllocator64<AP> PrimaryAllocator; -#else -// Currently, the 32-bit Sanitizer allocator has not yet benefited from all the -// security improvements brought to the 64-bit one. This makes the 32-bit -// version of Scudo slightly less toughened. -static const uptr RegionSizeLog = 20; -static const uptr NumRegions = SANITIZER_MMAP_RANGE_SIZE >> RegionSizeLog; -# if SANITIZER_WORDSIZE == 32 -typedef FlatByteMap<NumRegions> ByteMap; -# elif SANITIZER_WORDSIZE == 64 -typedef TwoLevelByteMap<(NumRegions >> 12), 1 << 12> ByteMap; -# endif // SANITIZER_WORDSIZE -typedef DefaultSizeClassMap SizeClassMap; -typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, 0, SizeClassMap, - RegionSizeLog, ByteMap> PrimaryAllocator; -#endif // SANITIZER_CAN_USE_ALLOCATOR64 - -typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache; -typedef ScudoLargeMmapAllocator SecondaryAllocator; -typedef CombinedAllocator<PrimaryAllocator, AllocatorCache, SecondaryAllocator> - ScudoAllocator; - -static ScudoAllocator &getAllocator(); - -static thread_local Xorshift128Plus Prng; // Global static cookie, initialized at start-up. static uptr Cookie; @@ -73,17 +35,32 @@ static uptr Cookie; // at compilation or at runtime. static atomic_uint8_t HashAlgorithm = { CRC32Software }; -// Helper function that will compute the chunk checksum, being passed all the -// the needed information as uptrs. It will opt for the hardware version of -// the checksumming function if available. -INLINE u32 hashUptrs(uptr Pointer, uptr *Array, uptr ArraySize, u8 HashType) { - u32 Crc; - Crc = computeCRC32(Cookie, Pointer, HashType); +INLINE u32 computeCRC32(uptr Crc, uptr Value, uptr *Array, uptr ArraySize) { + // If the hardware CRC32 feature is defined here, it was enabled everywhere, + // as opposed to only for scudo_crc32.cpp. This means that other hardware + // specific instructions were likely emitted at other places, and as a + // result there is no reason to not use it here. +#if defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) + Crc = CRC32_INTRINSIC(Crc, Value); + for (uptr i = 0; i < ArraySize; i++) + Crc = CRC32_INTRINSIC(Crc, Array[i]); + return Crc; +#else + if (atomic_load_relaxed(&HashAlgorithm) == CRC32Hardware) { + Crc = computeHardwareCRC32(Crc, Value); + for (uptr i = 0; i < ArraySize; i++) + Crc = computeHardwareCRC32(Crc, Array[i]); + return Crc; + } + Crc = computeSoftwareCRC32(Crc, Value); for (uptr i = 0; i < ArraySize; i++) - Crc = computeCRC32(Crc, Array[i], HashType); + Crc = computeSoftwareCRC32(Crc, Array[i]); return Crc; +#endif // defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) } +static ScudoBackendAllocator &getBackendAllocator(); + struct ScudoChunk : UnpackedHeader { // We can't use the offset member of the chunk itself, as we would double // fetch it without any warranty that it wouldn't have been tampered. To @@ -96,9 +73,11 @@ struct ScudoChunk : UnpackedHeader { // Returns the usable size for a chunk, meaning the amount of bytes from the // beginning of the user data to the end of the backend allocated chunk. uptr getUsableSize(UnpackedHeader *Header) { - uptr Size = getAllocator().GetActuallyAllocatedSize(getAllocBeg(Header)); + uptr Size = + getBackendAllocator().getActuallyAllocatedSize(getAllocBeg(Header), + Header->FromPrimary); if (Size == 0) - return Size; + return 0; return Size - AlignedChunkHeaderSize - (Header->Offset << MinAlignmentLog); } @@ -108,32 +87,43 @@ struct ScudoChunk : UnpackedHeader { ZeroChecksumHeader.Checksum = 0; uptr HeaderHolder[sizeof(UnpackedHeader) / sizeof(uptr)]; memcpy(&HeaderHolder, &ZeroChecksumHeader, sizeof(HeaderHolder)); - u32 Hash = hashUptrs(reinterpret_cast<uptr>(this), - HeaderHolder, - ARRAY_SIZE(HeaderHolder), - atomic_load_relaxed(&HashAlgorithm)); - return static_cast<u16>(Hash); + u32 Crc = computeCRC32(Cookie, reinterpret_cast<uptr>(this), HeaderHolder, + ARRAY_SIZE(HeaderHolder)); + return static_cast<u16>(Crc); } - // Checks the validity of a chunk by verifying its checksum. + // Checks the validity of a chunk by verifying its checksum. It doesn't + // incur termination in the event of an invalid chunk. bool isValid() { UnpackedHeader NewUnpackedHeader; const AtomicPackedHeader *AtomicHeader = reinterpret_cast<const AtomicPackedHeader *>(this); - PackedHeader NewPackedHeader = - AtomicHeader->load(std::memory_order_relaxed); + PackedHeader NewPackedHeader = atomic_load_relaxed(AtomicHeader); NewUnpackedHeader = bit_cast<UnpackedHeader>(NewPackedHeader); return (NewUnpackedHeader.Checksum == computeChecksum(&NewUnpackedHeader)); } + // Nulls out a chunk header. When returning the chunk to the backend, there + // is no need to store a valid ChunkAvailable header, as this would be + // computationally expensive. Zeroing out serves the same purpose by making + // the header invalid. In the extremely rare event where 0 would be a valid + // checksum for the chunk, the state of the chunk is ChunkAvailable anyway. + COMPILER_CHECK(ChunkAvailable == 0); + void eraseHeader() { + PackedHeader NullPackedHeader = 0; + AtomicPackedHeader *AtomicHeader = + reinterpret_cast<AtomicPackedHeader *>(this); + atomic_store_relaxed(AtomicHeader, NullPackedHeader); + } + // Loads and unpacks the header, verifying the checksum in the process. void loadHeader(UnpackedHeader *NewUnpackedHeader) const { const AtomicPackedHeader *AtomicHeader = reinterpret_cast<const AtomicPackedHeader *>(this); - PackedHeader NewPackedHeader = - AtomicHeader->load(std::memory_order_relaxed); + PackedHeader NewPackedHeader = atomic_load_relaxed(AtomicHeader); *NewUnpackedHeader = bit_cast<UnpackedHeader>(NewPackedHeader); - if (NewUnpackedHeader->Checksum != computeChecksum(NewUnpackedHeader)) { + if (UNLIKELY(NewUnpackedHeader->Checksum != + computeChecksum(NewUnpackedHeader))) { dieWithMessage("ERROR: corrupted chunk header at address %p\n", this); } } @@ -144,7 +134,7 @@ struct ScudoChunk : UnpackedHeader { PackedHeader NewPackedHeader = bit_cast<PackedHeader>(*NewUnpackedHeader); AtomicPackedHeader *AtomicHeader = reinterpret_cast<AtomicPackedHeader *>(this); - AtomicHeader->store(NewPackedHeader, std::memory_order_relaxed); + atomic_store_relaxed(AtomicHeader, NewPackedHeader); } // Packs and stores the header, computing the checksum in the process. We @@ -157,153 +147,156 @@ struct ScudoChunk : UnpackedHeader { PackedHeader OldPackedHeader = bit_cast<PackedHeader>(*OldUnpackedHeader); AtomicPackedHeader *AtomicHeader = reinterpret_cast<AtomicPackedHeader *>(this); - if (!AtomicHeader->compare_exchange_strong(OldPackedHeader, - NewPackedHeader, - std::memory_order_relaxed, - std::memory_order_relaxed)) { + if (UNLIKELY(!atomic_compare_exchange_strong(AtomicHeader, + &OldPackedHeader, + NewPackedHeader, + memory_order_relaxed))) { dieWithMessage("ERROR: race on chunk header at address %p\n", this); } } }; -static bool ScudoInitIsRunning = false; +ScudoChunk *getScudoChunk(uptr UserBeg) { + return reinterpret_cast<ScudoChunk *>(UserBeg - AlignedChunkHeaderSize); +} -static pthread_once_t GlobalInited = PTHREAD_ONCE_INIT; -static pthread_key_t PThreadKey; - -static thread_local bool ThreadInited = false; -static thread_local bool ThreadTornDown = false; -static thread_local AllocatorCache Cache; - -static void teardownThread(void *p) { - uptr v = reinterpret_cast<uptr>(p); - // The glibc POSIX thread-local-storage deallocation routine calls user - // provided destructors in a loop of PTHREAD_DESTRUCTOR_ITERATIONS. - // We want to be called last since other destructors might call free and the - // like, so we wait until PTHREAD_DESTRUCTOR_ITERATIONS before draining the - // quarantine and swallowing the cache. - if (v < PTHREAD_DESTRUCTOR_ITERATIONS) { - pthread_setspecific(PThreadKey, reinterpret_cast<void *>(v + 1)); - return; - } - drainQuarantine(); - getAllocator().DestroyCache(&Cache); - ThreadTornDown = true; +struct AllocatorOptions { + u32 QuarantineSizeMb; + u32 ThreadLocalQuarantineSizeKb; + bool MayReturnNull; + s32 ReleaseToOSIntervalMs; + bool DeallocationTypeMismatch; + bool DeleteSizeMismatch; + bool ZeroContents; + + void setFrom(const Flags *f, const CommonFlags *cf); + void copyTo(Flags *f, CommonFlags *cf) const; +}; + +void AllocatorOptions::setFrom(const Flags *f, const CommonFlags *cf) { + MayReturnNull = cf->allocator_may_return_null; + ReleaseToOSIntervalMs = cf->allocator_release_to_os_interval_ms; + QuarantineSizeMb = f->QuarantineSizeMb; + ThreadLocalQuarantineSizeKb = f->ThreadLocalQuarantineSizeKb; + DeallocationTypeMismatch = f->DeallocationTypeMismatch; + DeleteSizeMismatch = f->DeleteSizeMismatch; + ZeroContents = f->ZeroContents; +} + +void AllocatorOptions::copyTo(Flags *f, CommonFlags *cf) const { + cf->allocator_may_return_null = MayReturnNull; + cf->allocator_release_to_os_interval_ms = ReleaseToOSIntervalMs; + f->QuarantineSizeMb = QuarantineSizeMb; + f->ThreadLocalQuarantineSizeKb = ThreadLocalQuarantineSizeKb; + f->DeallocationTypeMismatch = DeallocationTypeMismatch; + f->DeleteSizeMismatch = DeleteSizeMismatch; + f->ZeroContents = ZeroContents; } -static void initInternal() { +static void initScudoInternal(const AllocatorOptions &Options); + +static bool ScudoInitIsRunning = false; + +void initScudo() { SanitizerToolName = "Scudo"; CHECK(!ScudoInitIsRunning && "Scudo init calls itself!"); ScudoInitIsRunning = true; - // Check is SSE4.2 is supported, if so, opt for the CRC32 hardware version. - if (testCPUFeature(CRC32CPUFeature)) { + // Check if hardware CRC32 is supported in the binary and by the platform, if + // so, opt for the CRC32 hardware version of the checksum. + if (computeHardwareCRC32 && testCPUFeature(CRC32CPUFeature)) atomic_store_relaxed(&HashAlgorithm, CRC32Hardware); - } initFlags(); AllocatorOptions Options; Options.setFrom(getFlags(), common_flags()); - initAllocator(Options); + initScudoInternal(Options); - MaybeStartBackgroudThread(); + // TODO(kostyak): determine if MaybeStartBackgroudThread could be of some use. ScudoInitIsRunning = false; } -static void initGlobal() { - pthread_key_create(&PThreadKey, teardownThread); - initInternal(); -} - -static void NOINLINE initThread() { - pthread_once(&GlobalInited, initGlobal); - pthread_setspecific(PThreadKey, reinterpret_cast<void *>(1)); - getAllocator().InitCache(&Cache); - ThreadInited = true; -} - struct QuarantineCallback { explicit QuarantineCallback(AllocatorCache *Cache) : Cache_(Cache) {} - // Chunk recycling function, returns a quarantined chunk to the backend. + // Chunk recycling function, returns a quarantined chunk to the backend, + // first making sure it hasn't been tampered with. void Recycle(ScudoChunk *Chunk) { UnpackedHeader Header; Chunk->loadHeader(&Header); - if (Header.State != ChunkQuarantine) { + if (UNLIKELY(Header.State != ChunkQuarantine)) { dieWithMessage("ERROR: invalid chunk state when recycling address %p\n", Chunk); } + Chunk->eraseHeader(); void *Ptr = Chunk->getAllocBeg(&Header); - getAllocator().Deallocate(Cache_, Ptr); + if (Header.FromPrimary) + getBackendAllocator().deallocatePrimary(Cache_, Ptr); + else + getBackendAllocator().deallocateSecondary(Ptr); } - /// Internal quarantine allocation and deallocation functions. + // Internal quarantine allocation and deallocation functions. We first check + // that the batches are indeed serviced by the Primary. + // TODO(kostyak): figure out the best way to protect the batches. + COMPILER_CHECK(sizeof(QuarantineBatch) < SizeClassMap::kMaxSize); void *Allocate(uptr Size) { - // The internal quarantine memory cannot be protected by us. But the only - // structures allocated are QuarantineBatch, that are 8KB for x64. So we - // will use mmap for those, and given that Deallocate doesn't pass a size - // in, we enforce the size of the allocation to be sizeof(QuarantineBatch). - // TODO(kostyak): switching to mmap impacts greatly performances, we have - // to find another solution - // CHECK_EQ(Size, sizeof(QuarantineBatch)); - // return MmapOrDie(Size, "QuarantineBatch"); - return getAllocator().Allocate(Cache_, Size, 1, false); + return getBackendAllocator().allocatePrimary(Cache_, Size); } void Deallocate(void *Ptr) { - // UnmapOrDie(Ptr, sizeof(QuarantineBatch)); - getAllocator().Deallocate(Cache_, Ptr); + getBackendAllocator().deallocatePrimary(Cache_, Ptr); } AllocatorCache *Cache_; }; typedef Quarantine<QuarantineCallback, ScudoChunk> ScudoQuarantine; -typedef ScudoQuarantine::Cache QuarantineCache; -static thread_local QuarantineCache ThreadQuarantineCache; +typedef ScudoQuarantine::Cache ScudoQuarantineCache; +COMPILER_CHECK(sizeof(ScudoQuarantineCache) <= + sizeof(ScudoThreadContext::QuarantineCachePlaceHolder)); -void AllocatorOptions::setFrom(const Flags *f, const CommonFlags *cf) { - MayReturnNull = cf->allocator_may_return_null; - ReleaseToOSIntervalMs = cf->allocator_release_to_os_interval_ms; - QuarantineSizeMb = f->QuarantineSizeMb; - ThreadLocalQuarantineSizeKb = f->ThreadLocalQuarantineSizeKb; - DeallocationTypeMismatch = f->DeallocationTypeMismatch; - DeleteSizeMismatch = f->DeleteSizeMismatch; - ZeroContents = f->ZeroContents; +AllocatorCache *getAllocatorCache(ScudoThreadContext *ThreadContext) { + return &ThreadContext->Cache; } -void AllocatorOptions::copyTo(Flags *f, CommonFlags *cf) const { - cf->allocator_may_return_null = MayReturnNull; - cf->allocator_release_to_os_interval_ms = ReleaseToOSIntervalMs; - f->QuarantineSizeMb = QuarantineSizeMb; - f->ThreadLocalQuarantineSizeKb = ThreadLocalQuarantineSizeKb; - f->DeallocationTypeMismatch = DeallocationTypeMismatch; - f->DeleteSizeMismatch = DeleteSizeMismatch; - f->ZeroContents = ZeroContents; +ScudoQuarantineCache *getQuarantineCache(ScudoThreadContext *ThreadContext) { + return reinterpret_cast< + ScudoQuarantineCache *>(ThreadContext->QuarantineCachePlaceHolder); } -struct Allocator { +ScudoPrng *getPrng(ScudoThreadContext *ThreadContext) { + return &ThreadContext->Prng; +} + +struct ScudoAllocator { static const uptr MaxAllowedMallocSize = FIRST_32_SECOND_64(2UL << 30, 1ULL << 40); - ScudoAllocator BackendAllocator; + typedef ReturnNullOrDieOnFailure FailureHandler; + + ScudoBackendAllocator BackendAllocator; ScudoQuarantine AllocatorQuarantine; + StaticSpinMutex GlobalPrngMutex; + ScudoPrng GlobalPrng; + // The fallback caches are used when the thread local caches have been // 'detroyed' on thread tear-down. They are protected by a Mutex as they can // be accessed by different threads. StaticSpinMutex FallbackMutex; AllocatorCache FallbackAllocatorCache; - QuarantineCache FallbackQuarantineCache; + ScudoQuarantineCache FallbackQuarantineCache; + ScudoPrng FallbackPrng; bool DeallocationTypeMismatch; bool ZeroContents; bool DeleteSizeMismatch; - explicit Allocator(LinkerInitialized) + explicit ScudoAllocator(LinkerInitialized) : AllocatorQuarantine(LINKER_INITIALIZED), FallbackQuarantineCache(LINKER_INITIALIZED) {} @@ -317,23 +310,23 @@ struct Allocator { // result, the maximum offset will be at most the maximum alignment for the // last size class minus the header size, in multiples of MinAlignment. UnpackedHeader Header = {}; - uptr MaxPrimaryAlignment = 1 << MostSignificantSetBitIndex( - SizeClassMap::kMaxSize - MinAlignment); - uptr MaxOffset = (MaxPrimaryAlignment - AlignedChunkHeaderSize) >> - MinAlignmentLog; + uptr MaxPrimaryAlignment = + 1 << MostSignificantSetBitIndex(SizeClassMap::kMaxSize - MinAlignment); + uptr MaxOffset = + (MaxPrimaryAlignment - AlignedChunkHeaderSize) >> MinAlignmentLog; Header.Offset = MaxOffset; if (Header.Offset != MaxOffset) { dieWithMessage("ERROR: the maximum possible offset doesn't fit in the " "header\n"); } - // Verify that we can fit the maximum amount of unused bytes in the header. - // Given that the Secondary fits the allocation to a page, the worst case - // scenario happens in the Primary. It will depend on the second to last - // and last class sizes, as well as the dynamic base for the Primary. The - // following is an over-approximation that works for our needs. - uptr MaxUnusedBytes = SizeClassMap::kMaxSize - 1 - AlignedChunkHeaderSize; - Header.UnusedBytes = MaxUnusedBytes; - if (Header.UnusedBytes != MaxUnusedBytes) { + // Verify that we can fit the maximum size or amount of unused bytes in the + // header. Given that the Secondary fits the allocation to a page, the worst + // case scenario happens in the Primary. It will depend on the second to + // last and last class sizes, as well as the dynamic base for the Primary. + // The following is an over-approximation that works for our needs. + uptr MaxSizeOrUnusedBytes = SizeClassMap::kMaxSize - 1; + Header.SizeOrUnusedBytes = MaxSizeOrUnusedBytes; + if (Header.SizeOrUnusedBytes != MaxSizeOrUnusedBytes) { dieWithMessage("ERROR: the maximum possible unused bytes doesn't fit in " "the header\n"); } @@ -341,182 +334,237 @@ struct Allocator { DeallocationTypeMismatch = Options.DeallocationTypeMismatch; DeleteSizeMismatch = Options.DeleteSizeMismatch; ZeroContents = Options.ZeroContents; - BackendAllocator.Init(Options.MayReturnNull, Options.ReleaseToOSIntervalMs); + SetAllocatorMayReturnNull(Options.MayReturnNull); + BackendAllocator.init(Options.ReleaseToOSIntervalMs); AllocatorQuarantine.Init( static_cast<uptr>(Options.QuarantineSizeMb) << 20, static_cast<uptr>(Options.ThreadLocalQuarantineSizeKb) << 10); - BackendAllocator.InitCache(&FallbackAllocatorCache); - Cookie = Prng.Next(); + GlobalPrng.init(); + Cookie = GlobalPrng.getU64(); + BackendAllocator.initCache(&FallbackAllocatorCache); + FallbackPrng.init(); } - // Helper function that checks for a valid Scudo chunk. + // Helper function that checks for a valid Scudo chunk. nullptr isn't. bool isValidPointer(const void *UserPtr) { - uptr ChunkBeg = reinterpret_cast<uptr>(UserPtr); - if (!IsAligned(ChunkBeg, MinAlignment)) { + initThreadMaybe(); + if (UNLIKELY(!UserPtr)) return false; - } - ScudoChunk *Chunk = - reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize); - return Chunk->isValid(); + uptr UserBeg = reinterpret_cast<uptr>(UserPtr); + if (!IsAligned(UserBeg, MinAlignment)) + return false; + return getScudoChunk(UserBeg)->isValid(); } // Allocates a chunk. - void *allocate(uptr Size, uptr Alignment, AllocType Type) { - if (UNLIKELY(!ThreadInited)) - initThread(); - if (!IsPowerOfTwo(Alignment)) { - dieWithMessage("ERROR: alignment is not a power of 2\n"); - } - if (Alignment > MaxAlignment) - return BackendAllocator.ReturnNullOrDieOnBadRequest(); - if (Alignment < MinAlignment) + void *allocate(uptr Size, uptr Alignment, AllocType Type, + bool ForceZeroContents = false) { + initThreadMaybe(); + if (UNLIKELY(Alignment > MaxAlignment)) + return FailureHandler::OnBadRequest(); + if (UNLIKELY(Alignment < MinAlignment)) Alignment = MinAlignment; - if (Size == 0) + if (UNLIKELY(Size >= MaxAllowedMallocSize)) + return FailureHandler::OnBadRequest(); + if (UNLIKELY(Size == 0)) Size = 1; - if (Size >= MaxAllowedMallocSize) - return BackendAllocator.ReturnNullOrDieOnBadRequest(); uptr NeededSize = RoundUpTo(Size, MinAlignment) + AlignedChunkHeaderSize; - if (Alignment > MinAlignment) - NeededSize += Alignment; - if (NeededSize >= MaxAllowedMallocSize) - return BackendAllocator.ReturnNullOrDieOnBadRequest(); - - // Primary backed and Secondary backed allocations have a different - // treatment. We deal with alignment requirements of Primary serviced - // allocations here, but the Secondary will take care of its own alignment - // needs, which means we also have to work around some limitations of the - // combined allocator to accommodate the situation. - bool FromPrimary = PrimaryAllocator::CanAllocate(NeededSize, MinAlignment); + uptr AlignedSize = (Alignment > MinAlignment) ? + NeededSize + (Alignment - AlignedChunkHeaderSize) : NeededSize; + if (UNLIKELY(AlignedSize >= MaxAllowedMallocSize)) + return FailureHandler::OnBadRequest(); + + // Primary and Secondary backed allocations have a different treatment. We + // deal with alignment requirements of Primary serviced allocations here, + // but the Secondary will take care of its own alignment needs. + bool FromPrimary = PrimaryAllocator::CanAllocate(AlignedSize, MinAlignment); void *Ptr; - if (LIKELY(!ThreadTornDown)) { - Ptr = BackendAllocator.Allocate(&Cache, NeededSize, - FromPrimary ? MinAlignment : Alignment); + u8 Salt; + uptr AllocSize; + if (FromPrimary) { + AllocSize = AlignedSize; + ScudoThreadContext *ThreadContext = getThreadContextAndLock(); + if (LIKELY(ThreadContext)) { + Salt = getPrng(ThreadContext)->getU8(); + Ptr = BackendAllocator.allocatePrimary(getAllocatorCache(ThreadContext), + AllocSize); + ThreadContext->unlock(); + } else { + SpinMutexLock l(&FallbackMutex); + Salt = FallbackPrng.getU8(); + Ptr = BackendAllocator.allocatePrimary(&FallbackAllocatorCache, + AllocSize); + } } else { - SpinMutexLock l(&FallbackMutex); - Ptr = BackendAllocator.Allocate(&FallbackAllocatorCache, NeededSize, - FromPrimary ? MinAlignment : Alignment); - } - if (!Ptr) - return BackendAllocator.ReturnNullOrDieOnOOM(); - - uptr AllocBeg = reinterpret_cast<uptr>(Ptr); - // If the allocation was serviced by the secondary, the returned pointer - // accounts for ChunkHeaderSize to pass the alignment check of the combined - // allocator. Adjust it here. - if (!FromPrimary) { - AllocBeg -= AlignedChunkHeaderSize; - if (Alignment > MinAlignment) - NeededSize -= Alignment; + { + SpinMutexLock l(&GlobalPrngMutex); + Salt = GlobalPrng.getU8(); + } + AllocSize = NeededSize; + Ptr = BackendAllocator.allocateSecondary(AllocSize, Alignment); } + if (UNLIKELY(!Ptr)) + return FailureHandler::OnOOM(); - uptr ActuallyAllocatedSize = BackendAllocator.GetActuallyAllocatedSize( - reinterpret_cast<void *>(AllocBeg)); // If requested, we will zero out the entire contents of the returned chunk. - if (ZeroContents && FromPrimary) - memset(Ptr, 0, ActuallyAllocatedSize); - - uptr ChunkBeg = AllocBeg + AlignedChunkHeaderSize; - if (!IsAligned(ChunkBeg, Alignment)) - ChunkBeg = RoundUpTo(ChunkBeg, Alignment); - CHECK_LE(ChunkBeg + Size, AllocBeg + NeededSize); - ScudoChunk *Chunk = - reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize); + if ((ForceZeroContents || ZeroContents) && FromPrimary) + memset(Ptr, 0, BackendAllocator.getActuallyAllocatedSize( + Ptr, /*FromPrimary=*/true)); + UnpackedHeader Header = {}; + uptr AllocBeg = reinterpret_cast<uptr>(Ptr); + uptr UserBeg = AllocBeg + AlignedChunkHeaderSize; + if (UNLIKELY(!IsAligned(UserBeg, Alignment))) { + // Since the Secondary takes care of alignment, a non-aligned pointer + // means it is from the Primary. It is also the only case where the offset + // field of the header would be non-zero. + CHECK(FromPrimary); + UserBeg = RoundUpTo(UserBeg, Alignment); + uptr Offset = UserBeg - AlignedChunkHeaderSize - AllocBeg; + Header.Offset = Offset >> MinAlignmentLog; + } + CHECK_LE(UserBeg + Size, AllocBeg + AllocSize); Header.State = ChunkAllocated; - uptr Offset = ChunkBeg - AlignedChunkHeaderSize - AllocBeg; - Header.Offset = Offset >> MinAlignmentLog; Header.AllocType = Type; - Header.UnusedBytes = ActuallyAllocatedSize - Offset - - AlignedChunkHeaderSize - Size; - Header.Salt = static_cast<u8>(Prng.Next()); - Chunk->storeHeader(&Header); - void *UserPtr = reinterpret_cast<void *>(ChunkBeg); - // TODO(kostyak): hooks sound like a terrible idea security wise but might - // be needed for things to work properly? + if (FromPrimary) { + Header.FromPrimary = 1; + Header.SizeOrUnusedBytes = Size; + } else { + // The secondary fits the allocations to a page, so the amount of unused + // bytes is the difference between the end of the user allocation and the + // next page boundary. + uptr PageSize = GetPageSizeCached(); + uptr TrailingBytes = (UserBeg + Size) & (PageSize - 1); + if (TrailingBytes) + Header.SizeOrUnusedBytes = PageSize - TrailingBytes; + } + Header.Salt = Salt; + getScudoChunk(UserBeg)->storeHeader(&Header); + void *UserPtr = reinterpret_cast<void *>(UserBeg); // if (&__sanitizer_malloc_hook) __sanitizer_malloc_hook(UserPtr, Size); return UserPtr; } + // Place a chunk in the quarantine. In the event of a zero-sized quarantine, + // we directly deallocate the chunk, otherwise the flow would lead to the + // chunk being loaded (and checked) twice, and stored (and checksummed) once, + // with no additional security value. + void quarantineOrDeallocateChunk(ScudoChunk *Chunk, UnpackedHeader *Header, + uptr Size) { + bool FromPrimary = Header->FromPrimary; + bool BypassQuarantine = (AllocatorQuarantine.GetCacheSize() == 0); + if (BypassQuarantine) { + Chunk->eraseHeader(); + void *Ptr = Chunk->getAllocBeg(Header); + if (FromPrimary) { + ScudoThreadContext *ThreadContext = getThreadContextAndLock(); + if (LIKELY(ThreadContext)) { + getBackendAllocator().deallocatePrimary( + getAllocatorCache(ThreadContext), Ptr); + ThreadContext->unlock(); + } else { + SpinMutexLock Lock(&FallbackMutex); + getBackendAllocator().deallocatePrimary(&FallbackAllocatorCache, Ptr); + } + } else { + getBackendAllocator().deallocateSecondary(Ptr); + } + } else { + UnpackedHeader NewHeader = *Header; + NewHeader.State = ChunkQuarantine; + Chunk->compareExchangeHeader(&NewHeader, Header); + ScudoThreadContext *ThreadContext = getThreadContextAndLock(); + if (LIKELY(ThreadContext)) { + AllocatorQuarantine.Put(getQuarantineCache(ThreadContext), + QuarantineCallback( + getAllocatorCache(ThreadContext)), + Chunk, Size); + ThreadContext->unlock(); + } else { + SpinMutexLock l(&FallbackMutex); + AllocatorQuarantine.Put(&FallbackQuarantineCache, + QuarantineCallback(&FallbackAllocatorCache), + Chunk, Size); + } + } + } + // Deallocates a Chunk, which means adding it to the delayed free list (or // Quarantine). void deallocate(void *UserPtr, uptr DeleteSize, AllocType Type) { - if (UNLIKELY(!ThreadInited)) - initThread(); - // TODO(kostyak): see hook comment above + initThreadMaybe(); // if (&__sanitizer_free_hook) __sanitizer_free_hook(UserPtr); - if (!UserPtr) + if (UNLIKELY(!UserPtr)) return; - uptr ChunkBeg = reinterpret_cast<uptr>(UserPtr); - if (!IsAligned(ChunkBeg, MinAlignment)) { + uptr UserBeg = reinterpret_cast<uptr>(UserPtr); + if (UNLIKELY(!IsAligned(UserBeg, MinAlignment))) { dieWithMessage("ERROR: attempted to deallocate a chunk not properly " "aligned at address %p\n", UserPtr); } - ScudoChunk *Chunk = - reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize); + ScudoChunk *Chunk = getScudoChunk(UserBeg); UnpackedHeader OldHeader; Chunk->loadHeader(&OldHeader); - if (OldHeader.State != ChunkAllocated) { + if (UNLIKELY(OldHeader.State != ChunkAllocated)) { dieWithMessage("ERROR: invalid chunk state when deallocating address " "%p\n", UserPtr); } - uptr UsableSize = Chunk->getUsableSize(&OldHeader); - UnpackedHeader NewHeader = OldHeader; - NewHeader.State = ChunkQuarantine; - Chunk->compareExchangeHeader(&NewHeader, &OldHeader); if (DeallocationTypeMismatch) { // The deallocation type has to match the allocation one. - if (NewHeader.AllocType != Type) { + if (OldHeader.AllocType != Type) { // With the exception of memalign'd Chunks, that can be still be free'd. - if (NewHeader.AllocType != FromMemalign || Type != FromMalloc) { + if (OldHeader.AllocType != FromMemalign || Type != FromMalloc) { dieWithMessage("ERROR: allocation type mismatch on address %p\n", - Chunk); + UserPtr); } } } - uptr Size = UsableSize - OldHeader.UnusedBytes; + uptr Size = OldHeader.FromPrimary ? OldHeader.SizeOrUnusedBytes : + Chunk->getUsableSize(&OldHeader) - OldHeader.SizeOrUnusedBytes; if (DeleteSizeMismatch) { if (DeleteSize && DeleteSize != Size) { dieWithMessage("ERROR: invalid sized delete on chunk at address %p\n", - Chunk); + UserPtr); } } - if (LIKELY(!ThreadTornDown)) { - AllocatorQuarantine.Put(&ThreadQuarantineCache, - QuarantineCallback(&Cache), Chunk, UsableSize); - } else { - SpinMutexLock l(&FallbackMutex); - AllocatorQuarantine.Put(&FallbackQuarantineCache, - QuarantineCallback(&FallbackAllocatorCache), - Chunk, UsableSize); - } + // If a small memory amount was allocated with a larger alignment, we want + // to take that into account. Otherwise the Quarantine would be filled with + // tiny chunks, taking a lot of VA memory. This is an approximation of the + // usable size, that allows us to not call GetActuallyAllocatedSize. + uptr LiableSize = Size + (OldHeader.Offset << MinAlignment); + quarantineOrDeallocateChunk(Chunk, &OldHeader, LiableSize); } // Reallocates a chunk. We can save on a new allocation if the new requested // size still fits in the chunk. void *reallocate(void *OldPtr, uptr NewSize) { - if (UNLIKELY(!ThreadInited)) - initThread(); - uptr ChunkBeg = reinterpret_cast<uptr>(OldPtr); - ScudoChunk *Chunk = - reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize); + initThreadMaybe(); + uptr UserBeg = reinterpret_cast<uptr>(OldPtr); + if (UNLIKELY(!IsAligned(UserBeg, MinAlignment))) { + dieWithMessage("ERROR: attempted to reallocate a chunk not properly " + "aligned at address %p\n", OldPtr); + } + ScudoChunk *Chunk = getScudoChunk(UserBeg); UnpackedHeader OldHeader; Chunk->loadHeader(&OldHeader); - if (OldHeader.State != ChunkAllocated) { + if (UNLIKELY(OldHeader.State != ChunkAllocated)) { dieWithMessage("ERROR: invalid chunk state when reallocating address " "%p\n", OldPtr); } - uptr Size = Chunk->getUsableSize(&OldHeader); - if (OldHeader.AllocType != FromMalloc) { + if (UNLIKELY(OldHeader.AllocType != FromMalloc)) { dieWithMessage("ERROR: invalid chunk type when reallocating address %p\n", - Chunk); + OldPtr); } - UnpackedHeader NewHeader = OldHeader; - // The new size still fits in the current chunk. - if (NewSize <= Size) { - NewHeader.UnusedBytes = Size - NewSize; + uptr UsableSize = Chunk->getUsableSize(&OldHeader); + // The new size still fits in the current chunk, and the size difference + // is reasonable. + if (NewSize <= UsableSize && + (UsableSize - NewSize) < (SizeClassMap::kMaxSize / 2)) { + UnpackedHeader NewHeader = OldHeader; + NewHeader.SizeOrUnusedBytes = + OldHeader.FromPrimary ? NewSize : UsableSize - NewSize; Chunk->compareExchangeHeader(&NewHeader, &OldHeader); return OldPtr; } @@ -524,36 +572,25 @@ struct Allocator { // old one. void *NewPtr = allocate(NewSize, MinAlignment, FromMalloc); if (NewPtr) { - uptr OldSize = Size - OldHeader.UnusedBytes; + uptr OldSize = OldHeader.FromPrimary ? OldHeader.SizeOrUnusedBytes : + UsableSize - OldHeader.SizeOrUnusedBytes; memcpy(NewPtr, OldPtr, Min(NewSize, OldSize)); - NewHeader.State = ChunkQuarantine; - Chunk->compareExchangeHeader(&NewHeader, &OldHeader); - if (LIKELY(!ThreadTornDown)) { - AllocatorQuarantine.Put(&ThreadQuarantineCache, - QuarantineCallback(&Cache), Chunk, Size); - } else { - SpinMutexLock l(&FallbackMutex); - AllocatorQuarantine.Put(&FallbackQuarantineCache, - QuarantineCallback(&FallbackAllocatorCache), - Chunk, Size); - } + quarantineOrDeallocateChunk(Chunk, &OldHeader, UsableSize); } return NewPtr; } // Helper function that returns the actual usable size of a chunk. uptr getUsableSize(const void *Ptr) { - if (UNLIKELY(!ThreadInited)) - initThread(); - if (!Ptr) + initThreadMaybe(); + if (UNLIKELY(!Ptr)) return 0; - uptr ChunkBeg = reinterpret_cast<uptr>(Ptr); - ScudoChunk *Chunk = - reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize); + uptr UserBeg = reinterpret_cast<uptr>(Ptr); + ScudoChunk *Chunk = getScudoChunk(UserBeg); UnpackedHeader Header; Chunk->loadHeader(&Header); // Getting the usable size of a chunk only makes sense if it's allocated. - if (Header.State != ChunkAllocated) { + if (UNLIKELY(Header.State != ChunkAllocated)) { dieWithMessage("ERROR: invalid chunk state when sizing address %p\n", Ptr); } @@ -561,40 +598,49 @@ struct Allocator { } void *calloc(uptr NMemB, uptr Size) { - if (UNLIKELY(!ThreadInited)) - initThread(); - uptr Total = NMemB * Size; - if (Size != 0 && Total / Size != NMemB) // Overflow check - return BackendAllocator.ReturnNullOrDieOnBadRequest(); - void *Ptr = allocate(Total, MinAlignment, FromMalloc); - // If ZeroContents, the content of the chunk has already been zero'd out. - if (!ZeroContents && Ptr && BackendAllocator.FromPrimary(Ptr)) - memset(Ptr, 0, getUsableSize(Ptr)); - return Ptr; + initThreadMaybe(); + if (UNLIKELY(CheckForCallocOverflow(NMemB, Size))) + return FailureHandler::OnBadRequest(); + return allocate(NMemB * Size, MinAlignment, FromMalloc, true); + } + + void commitBack(ScudoThreadContext *ThreadContext) { + AllocatorCache *Cache = getAllocatorCache(ThreadContext); + AllocatorQuarantine.Drain(getQuarantineCache(ThreadContext), + QuarantineCallback(Cache)); + BackendAllocator.destroyCache(Cache); } - void drainQuarantine() { - AllocatorQuarantine.Drain(&ThreadQuarantineCache, - QuarantineCallback(&Cache)); + uptr getStats(AllocatorStat StatType) { + initThreadMaybe(); + uptr stats[AllocatorStatCount]; + BackendAllocator.getStats(stats); + return stats[StatType]; } }; -static Allocator Instance(LINKER_INITIALIZED); +static ScudoAllocator Instance(LINKER_INITIALIZED); -static ScudoAllocator &getAllocator() { +static ScudoBackendAllocator &getBackendAllocator() { return Instance.BackendAllocator; } -void initAllocator(const AllocatorOptions &Options) { +static void initScudoInternal(const AllocatorOptions &Options) { Instance.init(Options); } -void drainQuarantine() { - Instance.drainQuarantine(); +void ScudoThreadContext::init() { + getBackendAllocator().initCache(&Cache); + Prng.init(); + memset(QuarantineCachePlaceHolder, 0, sizeof(QuarantineCachePlaceHolder)); +} + +void ScudoThreadContext::commitBack() { + Instance.commitBack(this); } void *scudoMalloc(uptr Size, AllocType Type) { - return Instance.allocate(Size, MinAlignment, Type); + return SetErrnoOnNull(Instance.allocate(Size, MinAlignment, Type)); } void scudoFree(void *Ptr, AllocType Type) { @@ -607,47 +653,56 @@ void scudoSizedFree(void *Ptr, uptr Size, AllocType Type) { void *scudoRealloc(void *Ptr, uptr Size) { if (!Ptr) - return Instance.allocate(Size, MinAlignment, FromMalloc); + return SetErrnoOnNull(Instance.allocate(Size, MinAlignment, FromMalloc)); if (Size == 0) { Instance.deallocate(Ptr, 0, FromMalloc); return nullptr; } - return Instance.reallocate(Ptr, Size); + return SetErrnoOnNull(Instance.reallocate(Ptr, Size)); } void *scudoCalloc(uptr NMemB, uptr Size) { - return Instance.calloc(NMemB, Size); + return SetErrnoOnNull(Instance.calloc(NMemB, Size)); } void *scudoValloc(uptr Size) { - return Instance.allocate(Size, GetPageSizeCached(), FromMemalign); -} - -void *scudoMemalign(uptr Alignment, uptr Size) { - return Instance.allocate(Size, Alignment, FromMemalign); + return SetErrnoOnNull( + Instance.allocate(Size, GetPageSizeCached(), FromMemalign)); } void *scudoPvalloc(uptr Size) { uptr PageSize = GetPageSizeCached(); - Size = RoundUpTo(Size, PageSize); - if (Size == 0) { - // pvalloc(0) should allocate one page. - Size = PageSize; + // pvalloc(0) should allocate one page. + Size = Size ? RoundUpTo(Size, PageSize) : PageSize; + return SetErrnoOnNull(Instance.allocate(Size, PageSize, FromMemalign)); +} + +void *scudoMemalign(uptr Alignment, uptr Size) { + if (UNLIKELY(!IsPowerOfTwo(Alignment))) { + errno = errno_EINVAL; + return ScudoAllocator::FailureHandler::OnBadRequest(); } - return Instance.allocate(Size, PageSize, FromMemalign); + return SetErrnoOnNull(Instance.allocate(Size, Alignment, FromMemalign)); } int scudoPosixMemalign(void **MemPtr, uptr Alignment, uptr Size) { - *MemPtr = Instance.allocate(Size, Alignment, FromMemalign); + if (UNLIKELY(!CheckPosixMemalignAlignment(Alignment))) { + ScudoAllocator::FailureHandler::OnBadRequest(); + return errno_EINVAL; + } + void *Ptr = Instance.allocate(Size, Alignment, FromMemalign); + if (UNLIKELY(!Ptr)) + return errno_ENOMEM; + *MemPtr = Ptr; return 0; } void *scudoAlignedAlloc(uptr Alignment, uptr Size) { - // size must be a multiple of the alignment. To avoid a division, we first - // make sure that alignment is a power of 2. - CHECK(IsPowerOfTwo(Alignment)); - CHECK_EQ((Size & (Alignment - 1)), 0); - return Instance.allocate(Size, Alignment, FromMalloc); + if (UNLIKELY(!CheckAlignedAllocAlignmentAndSize(Alignment, Size))) { + errno = errno_EINVAL; + return ScudoAllocator::FailureHandler::OnBadRequest(); + } + return SetErrnoOnNull(Instance.allocate(Size, Alignment, FromMalloc)); } uptr scudoMallocUsableSize(void *Ptr) { @@ -661,15 +716,11 @@ using namespace __scudo; // MallocExtension helper functions uptr __sanitizer_get_current_allocated_bytes() { - uptr stats[AllocatorStatCount]; - getAllocator().GetStats(stats); - return stats[AllocatorStatAllocated]; + return Instance.getStats(AllocatorStatAllocated); } uptr __sanitizer_get_heap_size() { - uptr stats[AllocatorStatCount]; - getAllocator().GetStats(stats); - return stats[AllocatorStatMapped]; + return Instance.getStats(AllocatorStatMapped); } uptr __sanitizer_get_free_bytes() { diff --git a/contrib/compiler-rt/lib/scudo/scudo_allocator.h b/contrib/compiler-rt/lib/scudo/scudo_allocator.h index 6431a2a..29d8599 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_allocator.h +++ b/contrib/compiler-rt/lib/scudo/scudo_allocator.h @@ -18,7 +18,9 @@ #include "sanitizer_common/sanitizer_allocator.h" -#include <atomic> +#if !SANITIZER_LINUX +# error "The Scudo hardened allocator is currently only supported on Linux." +#endif namespace __scudo { @@ -39,27 +41,28 @@ enum ChunkState : u8 { // using functions such as GetBlockBegin, that is fairly costly. Our first // implementation used the MetaData as well, which offers the advantage of // being stored away from the chunk itself, but accessing it was costly as -// well. The header will be atomically loaded and stored using the 16-byte -// primitives offered by the platform (likely requires cmpxchg16b support). +// well. The header will be atomically loaded and stored. typedef u64 PackedHeader; struct UnpackedHeader { - u64 Checksum : 16; - u64 UnusedBytes : 20; // Needed for reallocation purposes. - u64 State : 2; // available, allocated, or quarantined - u64 AllocType : 2; // malloc, new, new[], or memalign - u64 Offset : 16; // Offset from the beginning of the backend - // allocation to the beginning of the chunk itself, - // in multiples of MinAlignment. See comment about - // its maximum value and test in init(). - u64 Salt : 8; + u64 Checksum : 16; + u64 SizeOrUnusedBytes : 19; // Size for Primary backed allocations, amount of + // unused bytes in the chunk for Secondary ones. + u64 FromPrimary : 1; + u64 State : 2; // available, allocated, or quarantined + u64 AllocType : 2; // malloc, new, new[], or memalign + u64 Offset : 16; // Offset from the beginning of the backend + // allocation to the beginning of the chunk + // itself, in multiples of MinAlignment. See + // comment about its maximum value and in init(). + u64 Salt : 8; }; -typedef std::atomic<PackedHeader> AtomicPackedHeader; +typedef atomic_uint64_t AtomicPackedHeader; COMPILER_CHECK(sizeof(UnpackedHeader) == sizeof(PackedHeader)); // Minimum alignment of 8 bytes for 32-bit, 16 for 64-bit const uptr MinAlignmentLog = FIRST_32_SECOND_64(3, 4); -const uptr MaxAlignmentLog = 24; // 16 MB +const uptr MaxAlignmentLog = 24; // 16 MB const uptr MinAlignment = 1 << MinAlignmentLog; const uptr MaxAlignment = 1 << MaxAlignmentLog; @@ -67,21 +70,66 @@ const uptr ChunkHeaderSize = sizeof(PackedHeader); const uptr AlignedChunkHeaderSize = (ChunkHeaderSize + MinAlignment - 1) & ~(MinAlignment - 1); -struct AllocatorOptions { - u32 QuarantineSizeMb; - u32 ThreadLocalQuarantineSizeKb; - bool MayReturnNull; - s32 ReleaseToOSIntervalMs; - bool DeallocationTypeMismatch; - bool DeleteSizeMismatch; - bool ZeroContents; - - void setFrom(const Flags *f, const CommonFlags *cf); - void copyTo(Flags *f, CommonFlags *cf) const; +#if SANITIZER_CAN_USE_ALLOCATOR64 +const uptr AllocatorSpace = ~0ULL; +# if defined(__aarch64__) && SANITIZER_ANDROID +const uptr AllocatorSize = 0x4000000000ULL; // 256G. +# elif defined(__aarch64__) +const uptr AllocatorSize = 0x10000000000ULL; // 1T. +# else +const uptr AllocatorSize = 0x40000000000ULL; // 4T. +# endif +typedef DefaultSizeClassMap SizeClassMap; +struct AP64 { + static const uptr kSpaceBeg = AllocatorSpace; + static const uptr kSpaceSize = AllocatorSize; + static const uptr kMetadataSize = 0; + typedef __scudo::SizeClassMap SizeClassMap; + typedef NoOpMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = + SizeClassAllocator64FlagMasks::kRandomShuffleChunks; }; +typedef SizeClassAllocator64<AP64> PrimaryAllocator; +#else +// Currently, the 32-bit Sanitizer allocator has not yet benefited from all the +// security improvements brought to the 64-bit one. This makes the 32-bit +// version of Scudo slightly less toughened. +static const uptr RegionSizeLog = 20; +static const uptr NumRegions = SANITIZER_MMAP_RANGE_SIZE >> RegionSizeLog; +# if SANITIZER_WORDSIZE == 32 +typedef FlatByteMap<NumRegions> ByteMap; +# elif SANITIZER_WORDSIZE == 64 +typedef TwoLevelByteMap<(NumRegions >> 12), 1 << 12> ByteMap; +# endif // SANITIZER_WORDSIZE +typedef DefaultSizeClassMap SizeClassMap; +struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = 0; + typedef __scudo::SizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = RegionSizeLog; + typedef __scudo::ByteMap ByteMap; + typedef NoOpMapUnmapCallback MapUnmapCallback; + static const uptr kFlags = + SizeClassAllocator32FlagMasks::kRandomShuffleChunks; +}; +typedef SizeClassAllocator32<AP32> PrimaryAllocator; +#endif // SANITIZER_CAN_USE_ALLOCATOR64 + +// __sanitizer::RoundUp has a CHECK that is extraneous for us. Use our own. +INLINE uptr RoundUpTo(uptr Size, uptr Boundary) { + return (Size + Boundary - 1) & ~(Boundary - 1); +} + +#include "scudo_allocator_secondary.h" +#include "scudo_allocator_combined.h" -void initAllocator(const AllocatorOptions &options); -void drainQuarantine(); +typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache; +typedef ScudoLargeMmapAllocator SecondaryAllocator; +typedef ScudoCombinedAllocator<PrimaryAllocator, AllocatorCache, + SecondaryAllocator> ScudoBackendAllocator; + +void initScudo(); void *scudoMalloc(uptr Size, AllocType Type); void scudoFree(void *Ptr, AllocType Type); @@ -95,8 +143,6 @@ int scudoPosixMemalign(void **MemPtr, uptr Alignment, uptr Size); void *scudoAlignedAlloc(uptr Alignment, uptr Size); uptr scudoMallocUsableSize(void *Ptr); -#include "scudo_allocator_secondary.h" - } // namespace __scudo #endif // SCUDO_ALLOCATOR_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_allocator_combined.h b/contrib/compiler-rt/lib/scudo/scudo_allocator_combined.h new file mode 100644 index 0000000..7599c12 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_allocator_combined.h @@ -0,0 +1,76 @@ +//===-- scudo_allocator_combined.h ------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo Combined Allocator, dispatches allocation & deallocation requests to +/// the Primary or the Secondary backend allocators. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_ALLOCATOR_COMBINED_H_ +#define SCUDO_ALLOCATOR_COMBINED_H_ + +#ifndef SCUDO_ALLOCATOR_H_ +#error "This file must be included inside scudo_allocator.h." +#endif + +template <class PrimaryAllocator, class AllocatorCache, + class SecondaryAllocator> +class ScudoCombinedAllocator { + public: + void init(s32 ReleaseToOSIntervalMs) { + Primary.Init(ReleaseToOSIntervalMs); + Secondary.Init(); + Stats.Init(); + } + + // Primary allocations are always MinAlignment aligned, and as such do not + // require an Alignment parameter. + void *allocatePrimary(AllocatorCache *Cache, uptr Size) { + return Cache->Allocate(&Primary, Primary.ClassID(Size)); + } + + // Secondary allocations do not require a Cache, but do require an Alignment + // parameter. + void *allocateSecondary(uptr Size, uptr Alignment) { + return Secondary.Allocate(&Stats, Size, Alignment); + } + + void deallocatePrimary(AllocatorCache *Cache, void *Ptr) { + Cache->Deallocate(&Primary, Primary.GetSizeClass(Ptr), Ptr); + } + + void deallocateSecondary(void *Ptr) { + Secondary.Deallocate(&Stats, Ptr); + } + + uptr getActuallyAllocatedSize(void *Ptr, bool FromPrimary) { + if (FromPrimary) + return PrimaryAllocator::ClassIdToSize(Primary.GetSizeClass(Ptr)); + return Secondary.GetActuallyAllocatedSize(Ptr); + } + + void initCache(AllocatorCache *Cache) { + Cache->Init(&Stats); + } + + void destroyCache(AllocatorCache *Cache) { + Cache->Destroy(&Primary, &Stats); + } + + void getStats(AllocatorStatCounters StatType) const { + Stats.Get(StatType); + } + + private: + PrimaryAllocator Primary; + SecondaryAllocator Secondary; + AllocatorGlobalStats Stats; +}; + +#endif // SCUDO_ALLOCATOR_COMBINED_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_allocator_secondary.h b/contrib/compiler-rt/lib/scudo/scudo_allocator_secondary.h index b984f0d..dbfb225 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_allocator_secondary.h +++ b/contrib/compiler-rt/lib/scudo/scudo_allocator_secondary.h @@ -24,26 +24,24 @@ class ScudoLargeMmapAllocator { public: - void Init(bool AllocatorMayReturnNull) { + void Init() { PageSize = GetPageSizeCached(); - atomic_store(&MayReturnNull, AllocatorMayReturnNull, memory_order_relaxed); } void *Allocate(AllocatorStats *Stats, uptr Size, uptr Alignment) { + uptr UserSize = Size - AlignedChunkHeaderSize; // The Scudo frontend prevents us from allocating more than // MaxAllowedMallocSize, so integer overflow checks would be superfluous. uptr MapSize = Size + SecondaryHeaderSize; + if (Alignment > MinAlignment) + MapSize += Alignment; MapSize = RoundUpTo(MapSize, PageSize); // Account for 2 guard pages, one before and one after the chunk. MapSize += 2 * PageSize; - // The size passed to the Secondary comprises the alignment, if large - // enough. Subtract it here to get the requested size, including header. - if (Alignment > MinAlignment) - Size -= Alignment; uptr MapBeg = reinterpret_cast<uptr>(MmapNoAccess(MapSize)); if (MapBeg == ~static_cast<uptr>(0)) - return ReturnNullOrDieOnOOM(); + return ReturnNullOrDieOnFailure::OnOOM(); // A page-aligned pointer is assumed after that, so check it now. CHECK(IsAligned(MapBeg, PageSize)); uptr MapEnd = MapBeg + MapSize; @@ -51,32 +49,32 @@ class ScudoLargeMmapAllocator { // initial guard page, and both headers. This is the pointer that has to // abide by alignment requirements. uptr UserBeg = MapBeg + PageSize + HeadersSize; + uptr UserEnd = UserBeg + UserSize; // In the rare event of larger alignments, we will attempt to fit the mmap // area better and unmap extraneous memory. This will also ensure that the // offset and unused bytes field of the header stay small. if (Alignment > MinAlignment) { - if (UserBeg & (Alignment - 1)) - UserBeg += Alignment - (UserBeg & (Alignment - 1)); - CHECK_GE(UserBeg, MapBeg); - uptr NewMapBeg = RoundDownTo(UserBeg - HeadersSize, PageSize) - PageSize; - CHECK_GE(NewMapBeg, MapBeg); - uptr NewMapEnd = RoundUpTo(UserBeg + (Size - AlignedChunkHeaderSize), - PageSize) + PageSize; - CHECK_LE(NewMapEnd, MapEnd); - // Unmap the extra memory if it's large enough, on both sides. - uptr Diff = NewMapBeg - MapBeg; - if (Diff > PageSize) - UnmapOrDie(reinterpret_cast<void *>(MapBeg), Diff); - Diff = MapEnd - NewMapEnd; - if (Diff > PageSize) - UnmapOrDie(reinterpret_cast<void *>(NewMapEnd), Diff); - MapBeg = NewMapBeg; - MapEnd = NewMapEnd; - MapSize = NewMapEnd - NewMapBeg; + if (!IsAligned(UserBeg, Alignment)) { + UserBeg = RoundUpTo(UserBeg, Alignment); + CHECK_GE(UserBeg, MapBeg); + uptr NewMapBeg = RoundDownTo(UserBeg - HeadersSize, PageSize) - + PageSize; + CHECK_GE(NewMapBeg, MapBeg); + if (NewMapBeg != MapBeg) { + UnmapOrDie(reinterpret_cast<void *>(MapBeg), NewMapBeg - MapBeg); + MapBeg = NewMapBeg; + } + UserEnd = UserBeg + UserSize; + } + uptr NewMapEnd = RoundUpTo(UserEnd, PageSize) + PageSize; + if (NewMapEnd != MapEnd) { + UnmapOrDie(reinterpret_cast<void *>(NewMapEnd), MapEnd - NewMapEnd); + MapEnd = NewMapEnd; + } + MapSize = MapEnd - MapBeg; } - uptr UserEnd = UserBeg + (Size - AlignedChunkHeaderSize); CHECK_LE(UserEnd, MapEnd - PageSize); // Actually mmap the memory, preserving the guard pages on either side. CHECK_EQ(MapBeg + PageSize, reinterpret_cast<uptr>( @@ -88,83 +86,35 @@ class ScudoLargeMmapAllocator { // The primary adds the whole class size to the stats when allocating a // chunk, so we will do something similar here. But we will not account for // the guard pages. - Stats->Add(AllocatorStatAllocated, MapSize - 2 * PageSize); - Stats->Add(AllocatorStatMapped, MapSize - 2 * PageSize); - - return reinterpret_cast<void *>(UserBeg); - } - - void *ReturnNullOrDieOnBadRequest() { - if (atomic_load(&MayReturnNull, memory_order_acquire)) - return nullptr; - ReportAllocatorCannotReturnNull(false); - } - - void *ReturnNullOrDieOnOOM() { - if (atomic_load(&MayReturnNull, memory_order_acquire)) - return nullptr; - ReportAllocatorCannotReturnNull(true); - } + { + SpinMutexLock l(&StatsMutex); + Stats->Add(AllocatorStatAllocated, MapSize - 2 * PageSize); + Stats->Add(AllocatorStatMapped, MapSize - 2 * PageSize); + } - void SetMayReturnNull(bool AllocatorMayReturnNull) { - atomic_store(&MayReturnNull, AllocatorMayReturnNull, memory_order_release); + return reinterpret_cast<void *>(Ptr); } void Deallocate(AllocatorStats *Stats, void *Ptr) { SecondaryHeader *Header = getHeader(Ptr); - Stats->Sub(AllocatorStatAllocated, Header->MapSize - 2 * PageSize); - Stats->Sub(AllocatorStatMapped, Header->MapSize - 2 * PageSize); + { + SpinMutexLock l(&StatsMutex); + Stats->Sub(AllocatorStatAllocated, Header->MapSize - 2 * PageSize); + Stats->Sub(AllocatorStatMapped, Header->MapSize - 2 * PageSize); + } UnmapOrDie(reinterpret_cast<void *>(Header->MapBeg), Header->MapSize); } - uptr TotalMemoryUsed() { - UNIMPLEMENTED(); - } - - bool PointerIsMine(const void *Ptr) { - UNIMPLEMENTED(); - } - uptr GetActuallyAllocatedSize(void *Ptr) { SecondaryHeader *Header = getHeader(Ptr); - // Deduct PageSize as MapEnd includes the trailing guard page. + // Deduct PageSize as MapSize includes the trailing guard page. uptr MapEnd = Header->MapBeg + Header->MapSize - PageSize; return MapEnd - reinterpret_cast<uptr>(Ptr); } - void *GetMetaData(const void *Ptr) { - UNIMPLEMENTED(); - } - - void *GetBlockBegin(const void *Ptr) { - UNIMPLEMENTED(); - } - - void *GetBlockBeginFastLocked(void *Ptr) { - UNIMPLEMENTED(); - } - - void PrintStats() { - UNIMPLEMENTED(); - } - - void ForceLock() { - UNIMPLEMENTED(); - } - - void ForceUnlock() { - UNIMPLEMENTED(); - } - - void ForEachChunk(ForEachChunkCallback Callback, void *Arg) { - UNIMPLEMENTED(); - } - private: // A Secondary allocated chunk header contains the base of the mapping and - // its size. Currently, the base is always a page before the header, but - // we might want to extend that number in the future based on the size of - // the allocation. + // its size, which comprises the guard pages. struct SecondaryHeader { uptr MapBeg; uptr MapSize; @@ -182,7 +132,7 @@ class ScudoLargeMmapAllocator { const uptr SecondaryHeaderSize = sizeof(SecondaryHeader); const uptr HeadersSize = SecondaryHeaderSize + AlignedChunkHeaderSize; uptr PageSize; - atomic_uint8_t MayReturnNull; + SpinMutex StatsMutex; }; #endif // SCUDO_ALLOCATOR_SECONDARY_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_crc32.cpp b/contrib/compiler-rt/lib/scudo/scudo_crc32.cpp index 94c8c24..a267dc4 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_crc32.cpp +++ b/contrib/compiler-rt/lib/scudo/scudo_crc32.cpp @@ -12,42 +12,14 @@ /// //===----------------------------------------------------------------------===// -// Hardware CRC32 is supported at compilation via the following: -// - for i386 & x86_64: -msse4.2 -// - for ARM & AArch64: -march=armv8-a+crc or -mcrc -// An additional check must be performed at runtime as well to make sure the -// emitted instructions are valid on the target host. #include "scudo_crc32.h" -#include "scudo_utils.h" - -#if defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) -# ifdef __SSE4_2__ -# include <smmintrin.h> -# define CRC32_INTRINSIC FIRST_32_SECOND_64(_mm_crc32_u32, _mm_crc32_u64) -# endif -# ifdef __ARM_FEATURE_CRC32 -# include <arm_acle.h> -# define CRC32_INTRINSIC FIRST_32_SECOND_64(__crc32cw, __crc32cd) -# endif -#endif // defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) namespace __scudo { #if defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) -INLINE u32 computeHardwareCRC32(u32 Crc, uptr Data) { +u32 computeHardwareCRC32(u32 Crc, uptr Data) { return CRC32_INTRINSIC(Crc, Data); } - -u32 computeCRC32(u32 Crc, uptr Data, u8 HashType) { - if (HashType == CRC32Hardware) { - return computeHardwareCRC32(Crc, Data); - } - return computeSoftwareCRC32(Crc, Data); -} -#else -u32 computeCRC32(u32 Crc, uptr Data, u8 HashType) { - return computeSoftwareCRC32(Crc, Data); -} #endif // defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) } // namespace __scudo diff --git a/contrib/compiler-rt/lib/scudo/scudo_crc32.h b/contrib/compiler-rt/lib/scudo/scudo_crc32.h index 6635cc7..5ffcc62 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_crc32.h +++ b/contrib/compiler-rt/lib/scudo/scudo_crc32.h @@ -7,7 +7,7 @@ // //===----------------------------------------------------------------------===// /// -/// Header for scudo_crc32.cpp. +/// Scudo chunk header checksum related definitions. /// //===----------------------------------------------------------------------===// @@ -16,6 +16,23 @@ #include "sanitizer_common/sanitizer_internal_defs.h" +// Hardware CRC32 is supported at compilation via the following: +// - for i386 & x86_64: -msse4.2 +// - for ARM & AArch64: -march=armv8-a+crc or -mcrc +// An additional check must be performed at runtime as well to make sure the +// emitted instructions are valid on the target host. + +#if defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) +# ifdef __SSE4_2__ +# include <smmintrin.h> +# define CRC32_INTRINSIC FIRST_32_SECOND_64(_mm_crc32_u32, _mm_crc32_u64) +# endif +# ifdef __ARM_FEATURE_CRC32 +# include <arm_acle.h> +# define CRC32_INTRINSIC FIRST_32_SECOND_64(__crc32cw, __crc32cd) +# endif +#endif // defined(__SSE4_2__) || defined(__ARM_FEATURE_CRC32) + namespace __scudo { enum : u8 { @@ -23,7 +40,61 @@ enum : u8 { CRC32Hardware = 1, }; -u32 computeCRC32(u32 Crc, uptr Data, u8 HashType); +const static u32 CRC32Table[] = { + 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, + 0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, + 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 0x1db71064, 0x6ab020f2, + 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, + 0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, + 0xfa0f3d63, 0x8d080df5, 0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, + 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, 0x35b5a8fa, 0x42b2986c, + 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59, + 0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, + 0xcfba9599, 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, + 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 0x76dc4190, 0x01db7106, + 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433, + 0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, + 0x91646c97, 0xe6635c01, 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, + 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950, + 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65, + 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, + 0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, + 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa, + 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f, + 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, + 0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, + 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 0xe3630b12, 0x94643b84, + 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, + 0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, + 0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, + 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, 0xd6d6a3e8, 0xa1d1937e, + 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b, + 0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, + 0x316e8eef, 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, + 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe, 0xb2bd0b28, + 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d, + 0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, + 0x72076785, 0x05005713, 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, + 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242, + 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777, + 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, + 0x616bffd3, 0x166ccf45, 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, + 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc, + 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9, + 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, + 0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, + 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d +}; + +INLINE u32 computeSoftwareCRC32(u32 Crc, uptr Data) { + for (uptr i = 0; i < sizeof(Data); i++) { + Crc = CRC32Table[(Crc ^ Data) & 0xff] ^ (Crc >> 8); + Data >>= 8; + } + return Crc; +} + +SANITIZER_WEAK_ATTRIBUTE u32 computeHardwareCRC32(u32 Crc, uptr Data); } // namespace __scudo diff --git a/contrib/compiler-rt/lib/scudo/scudo_flags.cpp b/contrib/compiler-rt/lib/scudo/scudo_flags.cpp index b9c83810..90f0cbf 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_flags.cpp +++ b/contrib/compiler-rt/lib/scudo/scudo_flags.cpp @@ -68,7 +68,7 @@ void initFlags() { // Sanity checks and default settings for the Quarantine parameters. if (f->QuarantineSizeMb < 0) { - const int DefaultQuarantineSizeMb = 64; + const int DefaultQuarantineSizeMb = FIRST_32_SECOND_64(4, 16); f->QuarantineSizeMb = DefaultQuarantineSizeMb; } // We enforce an upper limit for the quarantine size of 4Gb. @@ -76,7 +76,8 @@ void initFlags() { dieWithMessage("ERROR: the quarantine size is too large\n"); } if (f->ThreadLocalQuarantineSizeKb < 0) { - const int DefaultThreadLocalQuarantineSizeKb = 1024; + const int DefaultThreadLocalQuarantineSizeKb = + FIRST_32_SECOND_64(64, 256); f->ThreadLocalQuarantineSizeKb = DefaultThreadLocalQuarantineSizeKb; } // And an upper limit of 128Mb for the thread quarantine cache. @@ -84,6 +85,10 @@ void initFlags() { dieWithMessage("ERROR: the per thread quarantine cache size is too " "large\n"); } + if (f->ThreadLocalQuarantineSizeKb == 0 && f->QuarantineSizeMb > 0) { + dieWithMessage("ERROR: ThreadLocalQuarantineSizeKb can be set to 0 only " + "when QuarantineSizeMb is set to 0\n"); + } } Flags *getFlags() { diff --git a/contrib/compiler-rt/lib/scudo/scudo_flags.inc b/contrib/compiler-rt/lib/scudo/scudo_flags.inc index c7a2acf..45f9ea8 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_flags.inc +++ b/contrib/compiler-rt/lib/scudo/scudo_flags.inc @@ -15,12 +15,14 @@ # error "Define SCUDO_FLAG prior to including this file!" #endif -SCUDO_FLAG(int, QuarantineSizeMb, 64, +// Default value is set in scudo_flags.cpp based on architecture. +SCUDO_FLAG(int, QuarantineSizeMb, -1, "Size (in Mb) of quarantine used to delay the actual deallocation " "of chunks. Lower value may reduce memory usage but decrease the " "effectiveness of the mitigation.") -SCUDO_FLAG(int, ThreadLocalQuarantineSizeKb, 1024, +// Default value is set in scudo_flags.cpp based on architecture. +SCUDO_FLAG(int, ThreadLocalQuarantineSizeKb, -1, "Size (in Kb) of per-thread cache used to offload the global " "quarantine. Lower value may reduce memory usage but might increase " "the contention on the global quarantine.") diff --git a/contrib/compiler-rt/lib/scudo/scudo_new_delete.cpp b/contrib/compiler-rt/lib/scudo/scudo_new_delete.cpp index c022bd0..cdefb12 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_new_delete.cpp +++ b/contrib/compiler-rt/lib/scudo/scudo_new_delete.cpp @@ -26,13 +26,18 @@ namespace std { struct nothrow_t {}; } // namespace std +// TODO(alekseys): throw std::bad_alloc instead of dying on OOM. CXX_OPERATOR_ATTRIBUTE void *operator new(size_t size) { - return scudoMalloc(size, FromNew); + void *res = scudoMalloc(size, FromNew); + if (UNLIKELY(!res)) DieOnFailure::OnOOM(); + return res; } CXX_OPERATOR_ATTRIBUTE void *operator new[](size_t size) { - return scudoMalloc(size, FromNewArray); + void *res = scudoMalloc(size, FromNewArray); + if (UNLIKELY(!res)) DieOnFailure::OnOOM(); + return res; } CXX_OPERATOR_ATTRIBUTE void *operator new(size_t size, std::nothrow_t const&) { diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls.h b/contrib/compiler-rt/lib/scudo/scudo_tls.h new file mode 100644 index 0000000..20c4920 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls.h @@ -0,0 +1,47 @@ +//===-- scudo_tls.h ---------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo thread local structure definition. +/// Implementation will differ based on the thread local storage primitives +/// offered by the underlying platform. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_TLS_H_ +#define SCUDO_TLS_H_ + +#include "scudo_allocator.h" +#include "scudo_utils.h" + +#include "sanitizer_common/sanitizer_linux.h" +#include "sanitizer_common/sanitizer_platform.h" + +namespace __scudo { + +// Platform specific base thread context definitions. +#include "scudo_tls_context_android.inc" +#include "scudo_tls_context_linux.inc" + +struct ALIGNED(64) ScudoThreadContext : public ScudoThreadContextPlatform { + AllocatorCache Cache; + ScudoPrng Prng; + uptr QuarantineCachePlaceHolder[4]; + void init(); + void commitBack(); +}; + +void initThread(); + +// Platform specific dastpath functions definitions. +#include "scudo_tls_android.inc" +#include "scudo_tls_linux.inc" + +} // namespace __scudo + +#endif // SCUDO_TLS_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_android.cpp b/contrib/compiler-rt/lib/scudo/scudo_tls_android.cpp new file mode 100644 index 0000000..ec74e37 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_android.cpp @@ -0,0 +1,95 @@ +//===-- scudo_tls_android.cpp -----------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo thread local structure implementation for Android. +/// +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" + +#if SANITIZER_LINUX && SANITIZER_ANDROID + +#include "scudo_tls.h" + +#include <pthread.h> + +namespace __scudo { + +static pthread_once_t GlobalInitialized = PTHREAD_ONCE_INIT; +static pthread_key_t PThreadKey; + +static atomic_uint32_t ThreadContextCurrentIndex; +static ScudoThreadContext *ThreadContexts; +static uptr NumberOfContexts; + +// sysconf(_SC_NPROCESSORS_{CONF,ONLN}) cannot be used as they allocate memory. +static uptr getNumberOfCPUs() { + cpu_set_t CPUs; + CHECK_EQ(sched_getaffinity(0, sizeof(cpu_set_t), &CPUs), 0); + return CPU_COUNT(&CPUs); +} + +static void initOnce() { + // Hack: TLS_SLOT_TSAN was introduced in N. To be able to use it on M for + // testing, we create an unused key. Since the key_data array follows the tls + // array, it basically gives us the extra entry we need. + // TODO(kostyak): remove and restrict to N and above. + CHECK_EQ(pthread_key_create(&PThreadKey, NULL), 0); + initScudo(); + NumberOfContexts = getNumberOfCPUs(); + ThreadContexts = reinterpret_cast<ScudoThreadContext *>( + MmapOrDie(sizeof(ScudoThreadContext) * NumberOfContexts, __func__)); + for (uptr i = 0; i < NumberOfContexts; i++) + ThreadContexts[i].init(); +} + +void initThread() { + pthread_once(&GlobalInitialized, initOnce); + // Initial context assignment is done in a plain round-robin fashion. + u32 Index = atomic_fetch_add(&ThreadContextCurrentIndex, 1, + memory_order_relaxed); + ScudoThreadContext *ThreadContext = + &ThreadContexts[Index % NumberOfContexts]; + *get_android_tls_ptr() = reinterpret_cast<uptr>(ThreadContext); +} + +ScudoThreadContext *getThreadContextAndLockSlow() { + ScudoThreadContext *ThreadContext; + // Go through all the contexts and find the first unlocked one. + for (u32 i = 0; i < NumberOfContexts; i++) { + ThreadContext = &ThreadContexts[i]; + if (ThreadContext->tryLock()) { + *get_android_tls_ptr() = reinterpret_cast<uptr>(ThreadContext); + return ThreadContext; + } + } + // No luck, find the one with the lowest precedence, and slow lock it. + u64 Precedence = UINT64_MAX; + for (u32 i = 0; i < NumberOfContexts; i++) { + u64 SlowLockPrecedence = ThreadContexts[i].getSlowLockPrecedence(); + if (SlowLockPrecedence && SlowLockPrecedence < Precedence) { + ThreadContext = &ThreadContexts[i]; + Precedence = SlowLockPrecedence; + } + } + if (LIKELY(Precedence != UINT64_MAX)) { + ThreadContext->lock(); + *get_android_tls_ptr() = reinterpret_cast<uptr>(ThreadContext); + return ThreadContext; + } + // Last resort (can this happen?), stick with the current one. + ThreadContext = + reinterpret_cast<ScudoThreadContext *>(*get_android_tls_ptr()); + ThreadContext->lock(); + return ThreadContext; +} + +} // namespace __scudo + +#endif // SANITIZER_LINUX && SANITIZER_ANDROID diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_android.inc b/contrib/compiler-rt/lib/scudo/scudo_tls_android.inc new file mode 100644 index 0000000..8ecad7a --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_android.inc @@ -0,0 +1,44 @@ +//===-- scudo_tls_android.inc -----------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo thread local structure fastpath functions implementation for Android. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_TLS_ANDROID_H_ +#define SCUDO_TLS_ANDROID_H_ + +#ifndef SCUDO_TLS_H_ +# error "This file must be included inside scudo_tls.h." +#endif // SCUDO_TLS_H_ + +#if SANITIZER_LINUX && SANITIZER_ANDROID + +ALWAYS_INLINE void initThreadMaybe() { + if (LIKELY(*get_android_tls_ptr())) + return; + initThread(); +} + +ScudoThreadContext *getThreadContextAndLockSlow(); + +ALWAYS_INLINE ScudoThreadContext *getThreadContextAndLock() { + ScudoThreadContext *ThreadContext = + reinterpret_cast<ScudoThreadContext *>(*get_android_tls_ptr()); + CHECK(ThreadContext); + // Try to lock the currently associated context. + if (ThreadContext->tryLock()) + return ThreadContext; + // If it failed, go the slow path. + return getThreadContextAndLockSlow(); +} + +#endif // SANITIZER_LINUX && SANITIZER_ANDROID + +#endif // SCUDO_TLS_ANDROID_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_context_android.inc b/contrib/compiler-rt/lib/scudo/scudo_tls_context_android.inc new file mode 100644 index 0000000..f195131 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_context_android.inc @@ -0,0 +1,54 @@ +//===-- scudo_tls_context_android.inc ---------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Android specific base thread context definition. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_TLS_CONTEXT_ANDROID_INC_ +#define SCUDO_TLS_CONTEXT_ANDROID_INC_ + +#ifndef SCUDO_TLS_H_ +# error "This file must be included inside scudo_tls.h." +#endif // SCUDO_TLS_H_ + +#if SANITIZER_LINUX && SANITIZER_ANDROID + +struct ScudoThreadContextPlatform { + INLINE bool tryLock() { + if (Mutex.TryLock()) { + atomic_store_relaxed(&SlowLockPrecedence, 0); + return true; + } + if (atomic_load_relaxed(&SlowLockPrecedence) == 0) + atomic_store_relaxed(&SlowLockPrecedence, NanoTime()); + return false; + } + + INLINE void lock() { + Mutex.Lock(); + atomic_store_relaxed(&SlowLockPrecedence, 0); + } + + INLINE void unlock() { + Mutex.Unlock(); + } + + INLINE u64 getSlowLockPrecedence() { + return atomic_load_relaxed(&SlowLockPrecedence); + } + + private: + StaticSpinMutex Mutex; + atomic_uint64_t SlowLockPrecedence; +}; + +#endif // SANITIZER_LINUX && SANITIZER_ANDROID + +#endif // SCUDO_TLS_CONTEXT_ANDROID_INC_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_context_linux.inc b/contrib/compiler-rt/lib/scudo/scudo_tls_context_linux.inc new file mode 100644 index 0000000..8d292bd --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_context_linux.inc @@ -0,0 +1,29 @@ +//===-- scudo_tls_context_linux.inc -----------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Linux specific base thread context definition. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_TLS_CONTEXT_LINUX_INC_ +#define SCUDO_TLS_CONTEXT_LINUX_INC_ + +#ifndef SCUDO_TLS_H_ +# error "This file must be included inside scudo_tls.h." +#endif // SCUDO_TLS_H_ + +#if SANITIZER_LINUX && !SANITIZER_ANDROID + +struct ScudoThreadContextPlatform { + ALWAYS_INLINE void unlock() {} +}; + +#endif // SANITIZER_LINUX && !SANITIZER_ANDROID + +#endif // SCUDO_TLS_CONTEXT_LINUX_INC_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_linux.cpp b/contrib/compiler-rt/lib/scudo/scudo_tls_linux.cpp new file mode 100644 index 0000000..1e38233 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_linux.cpp @@ -0,0 +1,66 @@ +//===-- scudo_tls_linux.cpp -------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo thread local structure implementation for platforms supporting +/// thread_local. +/// +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_platform.h" + +#if SANITIZER_LINUX && !SANITIZER_ANDROID + +#include "scudo_tls.h" + +#include <pthread.h> + +namespace __scudo { + +static pthread_once_t GlobalInitialized = PTHREAD_ONCE_INIT; +static pthread_key_t PThreadKey; + +__attribute__((tls_model("initial-exec"))) +THREADLOCAL ThreadState ScudoThreadState = ThreadNotInitialized; +__attribute__((tls_model("initial-exec"))) +THREADLOCAL ScudoThreadContext ThreadLocalContext; + +static void teardownThread(void *Ptr) { + uptr I = reinterpret_cast<uptr>(Ptr); + // The glibc POSIX thread-local-storage deallocation routine calls user + // provided destructors in a loop of PTHREAD_DESTRUCTOR_ITERATIONS. + // We want to be called last since other destructors might call free and the + // like, so we wait until PTHREAD_DESTRUCTOR_ITERATIONS before draining the + // quarantine and swallowing the cache. + if (I > 1) { + // If pthread_setspecific fails, we will go ahead with the teardown. + if (LIKELY(pthread_setspecific(PThreadKey, + reinterpret_cast<void *>(I - 1)) == 0)) + return; + } + ThreadLocalContext.commitBack(); + ScudoThreadState = ThreadTornDown; +} + + +static void initOnce() { + CHECK_EQ(pthread_key_create(&PThreadKey, teardownThread), 0); + initScudo(); +} + +void initThread() { + CHECK_EQ(pthread_once(&GlobalInitialized, initOnce), 0); + CHECK_EQ(pthread_setspecific(PThreadKey, reinterpret_cast<void *>( + GetPthreadDestructorIterations())), 0); + ThreadLocalContext.init(); + ScudoThreadState = ThreadInitialized; +} + +} // namespace __scudo + +#endif // SANITIZER_LINUX && !SANITIZER_ANDROID diff --git a/contrib/compiler-rt/lib/scudo/scudo_tls_linux.inc b/contrib/compiler-rt/lib/scudo/scudo_tls_linux.inc new file mode 100644 index 0000000..242ee33 --- /dev/null +++ b/contrib/compiler-rt/lib/scudo/scudo_tls_linux.inc @@ -0,0 +1,48 @@ +//===-- scudo_tls_linux.inc -------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +/// +/// Scudo thread local structure fastpath functions implementation for platforms +/// supporting thread_local. +/// +//===----------------------------------------------------------------------===// + +#ifndef SCUDO_TLS_LINUX_H_ +#define SCUDO_TLS_LINUX_H_ + +#ifndef SCUDO_TLS_H_ +# error "This file must be included inside scudo_tls.h." +#endif // SCUDO_TLS_H_ + +#if SANITIZER_LINUX && !SANITIZER_ANDROID + +enum ThreadState : u8 { + ThreadNotInitialized = 0, + ThreadInitialized, + ThreadTornDown, +}; +__attribute__((tls_model("initial-exec"))) +extern THREADLOCAL ThreadState ScudoThreadState; +__attribute__((tls_model("initial-exec"))) +extern THREADLOCAL ScudoThreadContext ThreadLocalContext; + +ALWAYS_INLINE void initThreadMaybe() { + if (LIKELY(ScudoThreadState != ThreadNotInitialized)) + return; + initThread(); +} + +ALWAYS_INLINE ScudoThreadContext *getThreadContextAndLock() { + if (UNLIKELY(ScudoThreadState == ThreadTornDown)) + return nullptr; + return &ThreadLocalContext; +} + +#endif // SANITIZER_LINUX && !SANITIZER_ANDROID + +#endif // SCUDO_TLS_LINUX_H_ diff --git a/contrib/compiler-rt/lib/scudo/scudo_utils.cpp b/contrib/compiler-rt/lib/scudo/scudo_utils.cpp index ffa65b2..f7903ff 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_utils.cpp +++ b/contrib/compiler-rt/lib/scudo/scudo_utils.cpp @@ -20,8 +20,9 @@ #if defined(__x86_64__) || defined(__i386__) # include <cpuid.h> #endif - -#include <cstring> +#if defined(__arm__) || defined(__aarch64__) +# include <sys/auxv.h> +#endif // TODO(kostyak): remove __sanitizer *Printf uses in favor for our own less // complicated string formatting code. The following is a @@ -82,12 +83,12 @@ CPUIDRegs getCPUFeatures() { } #ifndef bit_SSE4_2 -#define bit_SSE4_2 bit_SSE42 // clang and gcc have different defines. +# define bit_SSE4_2 bit_SSE42 // clang and gcc have different defines. #endif bool testCPUFeature(CPUFeature Feature) { - static CPUIDRegs FeaturesRegs = getCPUFeatures(); + CPUIDRegs FeaturesRegs = getCPUFeatures(); switch (Feature) { case CRC32CPUFeature: // CRC32 is provided by SSE 4.2. @@ -97,100 +98,29 @@ bool testCPUFeature(CPUFeature Feature) } return false; } -#else +#elif defined(__arm__) || defined(__aarch64__) +// For ARM and AArch64, hardware CRC32 support is indicated in the +// AT_HWVAL auxiliary vector. + +#ifndef HWCAP_CRC32 +# define HWCAP_CRC32 (1<<7) // HWCAP_CRC32 is missing on older platforms. +#endif + bool testCPUFeature(CPUFeature Feature) { - return false; -} -#endif // defined(__x86_64__) || defined(__i386__) + uptr HWCap = getauxval(AT_HWCAP); -// readRetry will attempt to read Count bytes from the Fd specified, and if -// interrupted will retry to read additional bytes to reach Count. -static ssize_t readRetry(int Fd, u8 *Buffer, size_t Count) { - ssize_t AmountRead = 0; - while (static_cast<size_t>(AmountRead) < Count) { - ssize_t Result = read(Fd, Buffer + AmountRead, Count - AmountRead); - if (Result > 0) - AmountRead += Result; - else if (!Result) - break; - else if (errno != EINTR) { - AmountRead = -1; + switch (Feature) { + case CRC32CPUFeature: + return !!(HWCap & HWCAP_CRC32); + default: break; - } } - return AmountRead; -} - -static void fillRandom(u8 *Data, ssize_t Size) { - int Fd = open("/dev/urandom", O_RDONLY); - if (Fd < 0) { - dieWithMessage("ERROR: failed to open /dev/urandom.\n"); - } - bool Success = readRetry(Fd, Data, Size) == Size; - close(Fd); - if (!Success) { - dieWithMessage("ERROR: failed to read enough data from /dev/urandom.\n"); - } -} - -// Default constructor for Xorshift128Plus seeds the state with /dev/urandom. -// TODO(kostyak): investigate using getrandom() if available. -Xorshift128Plus::Xorshift128Plus() { - fillRandom(reinterpret_cast<u8 *>(State), sizeof(State)); + return false; } - -const static u32 CRC32Table[] = { - 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, - 0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, - 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 0x1db71064, 0x6ab020f2, - 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, - 0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, - 0xfa0f3d63, 0x8d080df5, 0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, - 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, 0x35b5a8fa, 0x42b2986c, - 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59, - 0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, - 0xcfba9599, 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, - 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 0x76dc4190, 0x01db7106, - 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433, - 0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, - 0x91646c97, 0xe6635c01, 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, - 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950, - 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65, - 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, - 0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, - 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa, - 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f, - 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, - 0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, - 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 0xe3630b12, 0x94643b84, - 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, - 0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, - 0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, - 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, 0xd6d6a3e8, 0xa1d1937e, - 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b, - 0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, - 0x316e8eef, 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, - 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe, 0xb2bd0b28, - 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d, - 0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, - 0x72076785, 0x05005713, 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, - 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242, - 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777, - 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, - 0x616bffd3, 0x166ccf45, 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, - 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc, - 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9, - 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, - 0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, - 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d -}; - -u32 computeSoftwareCRC32(u32 Crc, uptr Data) { - for (uptr i = 0; i < sizeof(Data); i++) { - Crc = CRC32Table[(Crc ^ Data) & 0xff] ^ (Crc >> 8); - Data >>= 8; - } - return Crc; +#else +bool testCPUFeature(CPUFeature Feature) { + return false; } +#endif // defined(__x86_64__) || defined(__i386__) } // namespace __scudo diff --git a/contrib/compiler-rt/lib/scudo/scudo_utils.h b/contrib/compiler-rt/lib/scudo/scudo_utils.h index ef2a609..6c6c9d8 100644 --- a/contrib/compiler-rt/lib/scudo/scudo_utils.h +++ b/contrib/compiler-rt/lib/scudo/scudo_utils.h @@ -36,25 +36,57 @@ enum CPUFeature { }; bool testCPUFeature(CPUFeature feature); -// Tiny PRNG based on https://en.wikipedia.org/wiki/Xorshift#xorshift.2B -// The state (128 bits) will be stored in thread local storage. -struct Xorshift128Plus { +INLINE u64 rotl(const u64 X, int K) { + return (X << K) | (X >> (64 - K)); +} + +// XoRoShiRo128+ PRNG (http://xoroshiro.di.unimi.it/). +struct XoRoShiRo128Plus { public: - Xorshift128Plus(); - u64 Next() { - u64 x = State[0]; - const u64 y = State[1]; - State[0] = y; - x ^= x << 23; - State[1] = x ^ y ^ (x >> 17) ^ (y >> 26); - return State[1] + y; + void init() { + if (UNLIKELY(!GetRandom(reinterpret_cast<void *>(State), sizeof(State)))) { + // Early processes (eg: init) do not have /dev/urandom yet, but we still + // have to provide them with some degree of entropy. Not having a secure + // seed is not as problematic for them, as they are less likely to be + // the target of heap based vulnerabilities exploitation attempts. + State[0] = NanoTime(); + State[1] = 0; + } + fillCache(); } + u8 getU8() { + if (UNLIKELY(isCacheEmpty())) + fillCache(); + const u8 Result = static_cast<u8>(CachedBytes & 0xff); + CachedBytes >>= 8; + CachedBytesAvailable--; + return Result; + } + u64 getU64() { return next(); } + private: + u8 CachedBytesAvailable; + u64 CachedBytes; u64 State[2]; + u64 next() { + const u64 S0 = State[0]; + u64 S1 = State[1]; + const u64 Result = S0 + S1; + S1 ^= S0; + State[0] = rotl(S0, 55) ^ S1 ^ (S1 << 14); + State[1] = rotl(S1, 36); + return Result; + } + bool isCacheEmpty() { + return CachedBytesAvailable == 0; + } + void fillCache() { + CachedBytes = next(); + CachedBytesAvailable = sizeof(CachedBytes); + } }; -// Software CRC32 functions, to be used when hardware support is not detected. -u32 computeSoftwareCRC32(u32 Crc, uptr Data); +typedef XoRoShiRo128Plus ScudoPrng; } // namespace __scudo diff --git a/contrib/compiler-rt/lib/tsan/dd/dd_interceptors.cc b/contrib/compiler-rt/lib/tsan/dd/dd_interceptors.cc index 97c72dd..a39218f 100644 --- a/contrib/compiler-rt/lib/tsan/dd/dd_interceptors.cc +++ b/contrib/compiler-rt/lib/tsan/dd/dd_interceptors.cc @@ -270,20 +270,19 @@ namespace __dsan { static void InitDataSeg() { MemoryMappingLayout proc_maps(true); - uptr start, end, offset; char name[128]; + MemoryMappedSegment segment(name, ARRAY_SIZE(name)); bool prev_is_data = false; - while (proc_maps.Next(&start, &end, &offset, name, ARRAY_SIZE(name), - /*protection*/ 0)) { - bool is_data = offset != 0 && name[0] != 0; + while (proc_maps.Next(&segment)) { + bool is_data = segment.offset != 0 && segment.filename[0] != 0; // BSS may get merged with [heap] in /proc/self/maps. This is not very // reliable. - bool is_bss = offset == 0 && - (name[0] == 0 || internal_strcmp(name, "[heap]") == 0) && prev_is_data; - if (g_data_start == 0 && is_data) - g_data_start = start; - if (is_bss) - g_data_end = end; + bool is_bss = segment.offset == 0 && + (segment.filename[0] == 0 || + internal_strcmp(segment.filename, "[heap]") == 0) && + prev_is_data; + if (g_data_start == 0 && is_data) g_data_start = segment.start; + if (is_bss) g_data_end = segment.end; prev_is_data = is_data; } VPrintf(1, "guessed data_start=%p data_end=%p\n", g_data_start, g_data_end); diff --git a/contrib/compiler-rt/lib/tsan/go/tsan_go.cc b/contrib/compiler-rt/lib/tsan/go/tsan_go.cc index 34625c8..d7a9e0b 100644 --- a/contrib/compiler-rt/lib/tsan/go/tsan_go.cc +++ b/contrib/compiler-rt/lib/tsan/go/tsan_go.cc @@ -214,7 +214,7 @@ void __tsan_go_start(ThreadState *parent, ThreadState **pthr, void *pc) { ThreadState *thr = AllocGoroutine(); *pthr = thr; int goid = ThreadCreate(parent, (uptr)pc, 0, true); - ThreadStart(thr, goid, 0); + ThreadStart(thr, goid, 0, /*workerthread*/ false); } void __tsan_go_end(ThreadState *thr) { @@ -247,13 +247,17 @@ void __tsan_finalizer_goroutine(ThreadState *thr) { } void __tsan_mutex_before_lock(ThreadState *thr, uptr addr, uptr write) { + if (write) + MutexPreLock(thr, 0, addr); + else + MutexPreReadLock(thr, 0, addr); } void __tsan_mutex_after_lock(ThreadState *thr, uptr addr, uptr write) { if (write) - MutexLock(thr, 0, addr); + MutexPostLock(thr, 0, addr); else - MutexReadLock(thr, 0, addr); + MutexPostReadLock(thr, 0, addr); } void __tsan_mutex_before_unlock(ThreadState *thr, uptr addr, uptr write) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan.syms.extra b/contrib/compiler-rt/lib/tsan/rtl/tsan.syms.extra index 22dfde9..ab5b5a4 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan.syms.extra +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan.syms.extra @@ -9,6 +9,16 @@ __tsan_java* __tsan_unaligned* __tsan_release __tsan_acquire +__tsan_mutex_create +__tsan_mutex_destroy +__tsan_mutex_pre_lock +__tsan_mutex_post_lock +__tsan_mutex_pre_unlock +__tsan_mutex_post_unlock +__tsan_mutex_pre_signal +__tsan_mutex_post_signal +__tsan_mutex_pre_divert +__tsan_mutex_post_divert __ubsan_* Annotate* WTFAnnotate* diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.cc index 32435ad..ef984a4 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.cc @@ -61,20 +61,13 @@ // an exclusive lock; ThreadClock's are private to respective threads and so // do not need any protection. // -// Description of ThreadClock state: -// clk_ - fixed size vector clock. -// nclk_ - effective size of the vector clock (the rest is zeros). -// tid_ - index of the thread associated with he clock ("current thread"). -// last_acquire_ - current thread time when it acquired something from -// other threads. -// // Description of SyncClock state: // clk_ - variable size vector clock, low kClkBits hold timestamp, // the remaining bits hold "acquired" flag (the actual value is thread's // reused counter); // if acquried == thr->reused_, then the respective thread has already -// acquired this clock (except possibly dirty_tids_). -// dirty_tids_ - holds up to two indeces in the vector clock that other threads +// acquired this clock (except possibly for dirty elements). +// dirty_ - holds up to two indeces in the vector clock that other threads // need to acquire regardless of "acquired" flag value; // release_store_tid_ - denotes that the clock state is a result of // release-store operation by the thread with release_store_tid_ index. @@ -90,18 +83,51 @@ namespace __tsan { +static atomic_uint32_t *ref_ptr(ClockBlock *cb) { + return reinterpret_cast<atomic_uint32_t *>(&cb->table[ClockBlock::kRefIdx]); +} + +// Drop reference to the first level block idx. +static void UnrefClockBlock(ClockCache *c, u32 idx, uptr blocks) { + ClockBlock *cb = ctx->clock_alloc.Map(idx); + atomic_uint32_t *ref = ref_ptr(cb); + u32 v = atomic_load(ref, memory_order_acquire); + for (;;) { + CHECK_GT(v, 0); + if (v == 1) + break; + if (atomic_compare_exchange_strong(ref, &v, v - 1, memory_order_acq_rel)) + return; + } + // First level block owns second level blocks, so them as well. + for (uptr i = 0; i < blocks; i++) + ctx->clock_alloc.Free(c, cb->table[ClockBlock::kBlockIdx - i]); + ctx->clock_alloc.Free(c, idx); +} + ThreadClock::ThreadClock(unsigned tid, unsigned reused) : tid_(tid) - , reused_(reused + 1) { // 0 has special meaning + , reused_(reused + 1) // 0 has special meaning + , cached_idx_() + , cached_size_() + , cached_blocks_() { CHECK_LT(tid, kMaxTidInClock); CHECK_EQ(reused_, ((u64)reused_ << kClkBits) >> kClkBits); nclk_ = tid_ + 1; last_acquire_ = 0; internal_memset(clk_, 0, sizeof(clk_)); - clk_[tid_].reused = reused_; } -void ThreadClock::acquire(ClockCache *c, const SyncClock *src) { +void ThreadClock::ResetCached(ClockCache *c) { + if (cached_idx_) { + UnrefClockBlock(c, cached_idx_, cached_blocks_); + cached_idx_ = 0; + cached_size_ = 0; + cached_blocks_ = 0; + } +} + +void ThreadClock::acquire(ClockCache *c, SyncClock *src) { DCHECK_LE(nclk_, kMaxTid); DCHECK_LE(src->size_, kMaxTid); CPP_STAT_INC(StatClockAcquire); @@ -113,52 +139,46 @@ void ThreadClock::acquire(ClockCache *c, const SyncClock *src) { return; } - // Check if we've already acquired src after the last release operation on src bool acquired = false; - if (nclk > tid_) { - CPP_STAT_INC(StatClockAcquireLarge); - if (src->elem(tid_).reused == reused_) { - CPP_STAT_INC(StatClockAcquireRepeat); - for (unsigned i = 0; i < kDirtyTids; i++) { - unsigned tid = src->dirty_tids_[i]; - if (tid != kInvalidTid) { - u64 epoch = src->elem(tid).epoch; - if (clk_[tid].epoch < epoch) { - clk_[tid].epoch = epoch; - acquired = true; - } - } - } - if (acquired) { - CPP_STAT_INC(StatClockAcquiredSomething); - last_acquire_ = clk_[tid_].epoch; + for (unsigned i = 0; i < kDirtyTids; i++) { + SyncClock::Dirty dirty = src->dirty_[i]; + unsigned tid = dirty.tid; + if (tid != kInvalidTid) { + if (clk_[tid] < dirty.epoch) { + clk_[tid] = dirty.epoch; + acquired = true; } - return; } } - // O(N) acquire. - CPP_STAT_INC(StatClockAcquireFull); - nclk_ = max(nclk_, nclk); - for (uptr i = 0; i < nclk; i++) { - u64 epoch = src->elem(i).epoch; - if (clk_[i].epoch < epoch) { - clk_[i].epoch = epoch; - acquired = true; + // Check if we've already acquired src after the last release operation on src + if (tid_ >= nclk || src->elem(tid_).reused != reused_) { + // O(N) acquire. + CPP_STAT_INC(StatClockAcquireFull); + nclk_ = max(nclk_, nclk); + u64 *dst_pos = &clk_[0]; + for (ClockElem &src_elem : *src) { + u64 epoch = src_elem.epoch; + if (*dst_pos < epoch) { + *dst_pos = epoch; + acquired = true; + } + dst_pos++; } - } - // Remember that this thread has acquired this clock. - if (nclk > tid_) - src->elem(tid_).reused = reused_; + // Remember that this thread has acquired this clock. + if (nclk > tid_) + src->elem(tid_).reused = reused_; + } if (acquired) { CPP_STAT_INC(StatClockAcquiredSomething); - last_acquire_ = clk_[tid_].epoch; + last_acquire_ = clk_[tid_]; + ResetCached(c); } } -void ThreadClock::release(ClockCache *c, SyncClock *dst) const { +void ThreadClock::release(ClockCache *c, SyncClock *dst) { DCHECK_LE(nclk_, kMaxTid); DCHECK_LE(dst->size_, kMaxTid); @@ -178,7 +198,7 @@ void ThreadClock::release(ClockCache *c, SyncClock *dst) const { // since the last release on dst. If so, we need to update // only dst->elem(tid_). if (dst->elem(tid_).epoch > last_acquire_) { - UpdateCurrentThread(dst); + UpdateCurrentThread(c, dst); if (dst->release_store_tid_ != tid_ || dst->release_store_reused_ != reused_) dst->release_store_tid_ = kInvalidTid; @@ -187,23 +207,24 @@ void ThreadClock::release(ClockCache *c, SyncClock *dst) const { // O(N) release. CPP_STAT_INC(StatClockReleaseFull); + dst->Unshare(c); // First, remember whether we've acquired dst. bool acquired = IsAlreadyAcquired(dst); if (acquired) CPP_STAT_INC(StatClockReleaseAcquired); // Update dst->clk_. - for (uptr i = 0; i < nclk_; i++) { - ClockElem &ce = dst->elem(i); - ce.epoch = max(ce.epoch, clk_[i].epoch); + dst->FlushDirty(); + uptr i = 0; + for (ClockElem &ce : *dst) { + ce.epoch = max(ce.epoch, clk_[i]); ce.reused = 0; + i++; } // Clear 'acquired' flag in the remaining elements. if (nclk_ < dst->size_) CPP_STAT_INC(StatClockReleaseClearTail); for (uptr i = nclk_; i < dst->size_; i++) dst->elem(i).reused = 0; - for (unsigned i = 0; i < kDirtyTids; i++) - dst->dirty_tids_[i] = kInvalidTid; dst->release_store_tid_ = kInvalidTid; dst->release_store_reused_ = 0; // If we've acquired dst, remember this fact, @@ -212,11 +233,37 @@ void ThreadClock::release(ClockCache *c, SyncClock *dst) const { dst->elem(tid_).reused = reused_; } -void ThreadClock::ReleaseStore(ClockCache *c, SyncClock *dst) const { +void ThreadClock::ReleaseStore(ClockCache *c, SyncClock *dst) { DCHECK_LE(nclk_, kMaxTid); DCHECK_LE(dst->size_, kMaxTid); CPP_STAT_INC(StatClockStore); + if (dst->size_ == 0 && cached_idx_ != 0) { + // Reuse the cached clock. + // Note: we could reuse/cache the cached clock in more cases: + // we could update the existing clock and cache it, or replace it with the + // currently cached clock and release the old one. And for a shared + // existing clock, we could replace it with the currently cached; + // or unshare, update and cache. But, for simplicity, we currnetly reuse + // cached clock only when the target clock is empty. + dst->tab_ = ctx->clock_alloc.Map(cached_idx_); + dst->tab_idx_ = cached_idx_; + dst->size_ = cached_size_; + dst->blocks_ = cached_blocks_; + CHECK_EQ(dst->dirty_[0].tid, kInvalidTid); + // The cached clock is shared (immutable), + // so this is where we store the current clock. + dst->dirty_[0].tid = tid_; + dst->dirty_[0].epoch = clk_[tid_]; + dst->release_store_tid_ = tid_; + dst->release_store_reused_ = reused_; + // Rememeber that we don't need to acquire it in future. + dst->elem(tid_).reused = reused_; + // Grab a reference. + atomic_fetch_add(ref_ptr(dst->tab_), 1, memory_order_relaxed); + return; + } + // Check if we need to resize dst. if (dst->size_ < nclk_) dst->Resize(c, nclk_); @@ -225,32 +272,41 @@ void ThreadClock::ReleaseStore(ClockCache *c, SyncClock *dst) const { dst->release_store_reused_ == reused_ && dst->elem(tid_).epoch > last_acquire_) { CPP_STAT_INC(StatClockStoreFast); - UpdateCurrentThread(dst); + UpdateCurrentThread(c, dst); return; } // O(N) release-store. CPP_STAT_INC(StatClockStoreFull); - for (uptr i = 0; i < nclk_; i++) { - ClockElem &ce = dst->elem(i); - ce.epoch = clk_[i].epoch; + dst->Unshare(c); + // Note: dst can be larger than this ThreadClock. + // This is fine since clk_ beyond size is all zeros. + uptr i = 0; + for (ClockElem &ce : *dst) { + ce.epoch = clk_[i]; ce.reused = 0; + i++; } - // Clear the tail of dst->clk_. - if (nclk_ < dst->size_) { - for (uptr i = nclk_; i < dst->size_; i++) { - ClockElem &ce = dst->elem(i); - ce.epoch = 0; - ce.reused = 0; - } - CPP_STAT_INC(StatClockStoreTail); - } - for (unsigned i = 0; i < kDirtyTids; i++) - dst->dirty_tids_[i] = kInvalidTid; + for (uptr i = 0; i < kDirtyTids; i++) + dst->dirty_[i].tid = kInvalidTid; dst->release_store_tid_ = tid_; dst->release_store_reused_ = reused_; // Rememeber that we don't need to acquire it in future. dst->elem(tid_).reused = reused_; + + // If the resulting clock is cachable, cache it for future release operations. + // The clock is always cachable if we released to an empty sync object. + if (cached_idx_ == 0 && dst->Cachable()) { + // Grab a reference to the ClockBlock. + atomic_uint32_t *ref = ref_ptr(dst->tab_); + if (atomic_load(ref, memory_order_acquire) == 1) + atomic_store_relaxed(ref, 2); + else + atomic_fetch_add(ref_ptr(dst->tab_), 1, memory_order_relaxed); + cached_idx_ = dst->tab_idx_; + cached_size_ = dst->size_; + cached_blocks_ = dst->blocks_; + } } void ThreadClock::acq_rel(ClockCache *c, SyncClock *dst) { @@ -260,157 +316,248 @@ void ThreadClock::acq_rel(ClockCache *c, SyncClock *dst) { } // Updates only single element related to the current thread in dst->clk_. -void ThreadClock::UpdateCurrentThread(SyncClock *dst) const { +void ThreadClock::UpdateCurrentThread(ClockCache *c, SyncClock *dst) const { // Update the threads time, but preserve 'acquired' flag. - dst->elem(tid_).epoch = clk_[tid_].epoch; - for (unsigned i = 0; i < kDirtyTids; i++) { - if (dst->dirty_tids_[i] == tid_) { - CPP_STAT_INC(StatClockReleaseFast1); - return; - } - if (dst->dirty_tids_[i] == kInvalidTid) { - CPP_STAT_INC(StatClockReleaseFast2); - dst->dirty_tids_[i] = tid_; + SyncClock::Dirty *dirty = &dst->dirty_[i]; + const unsigned tid = dirty->tid; + if (tid == tid_ || tid == kInvalidTid) { + CPP_STAT_INC(StatClockReleaseFast); + dirty->tid = tid_; + dirty->epoch = clk_[tid_]; return; } } // Reset all 'acquired' flags, O(N). + // We are going to touch dst elements, so we need to unshare it. + dst->Unshare(c); CPP_STAT_INC(StatClockReleaseSlow); + dst->elem(tid_).epoch = clk_[tid_]; for (uptr i = 0; i < dst->size_; i++) dst->elem(i).reused = 0; - for (unsigned i = 0; i < kDirtyTids; i++) - dst->dirty_tids_[i] = kInvalidTid; + dst->FlushDirty(); } -// Checks whether the current threads has already acquired src. +// Checks whether the current thread has already acquired src. bool ThreadClock::IsAlreadyAcquired(const SyncClock *src) const { if (src->elem(tid_).reused != reused_) return false; for (unsigned i = 0; i < kDirtyTids; i++) { - unsigned tid = src->dirty_tids_[i]; - if (tid != kInvalidTid) { - if (clk_[tid].epoch < src->elem(tid).epoch) + SyncClock::Dirty dirty = src->dirty_[i]; + if (dirty.tid != kInvalidTid) { + if (clk_[dirty.tid] < dirty.epoch) return false; } } return true; } +// Sets a single element in the vector clock. +// This function is called only from weird places like AcquireGlobal. +void ThreadClock::set(ClockCache *c, unsigned tid, u64 v) { + DCHECK_LT(tid, kMaxTid); + DCHECK_GE(v, clk_[tid]); + clk_[tid] = v; + if (nclk_ <= tid) + nclk_ = tid + 1; + last_acquire_ = clk_[tid_]; + ResetCached(c); +} + +void ThreadClock::DebugDump(int(*printf)(const char *s, ...)) { + printf("clock=["); + for (uptr i = 0; i < nclk_; i++) + printf("%s%llu", i == 0 ? "" : ",", clk_[i]); + printf("] tid=%u/%u last_acq=%llu", tid_, reused_, last_acquire_); +} + +SyncClock::SyncClock() { + ResetImpl(); +} + +SyncClock::~SyncClock() { + // Reset must be called before dtor. + CHECK_EQ(size_, 0); + CHECK_EQ(blocks_, 0); + CHECK_EQ(tab_, 0); + CHECK_EQ(tab_idx_, 0); +} + +void SyncClock::Reset(ClockCache *c) { + if (size_) + UnrefClockBlock(c, tab_idx_, blocks_); + ResetImpl(); +} + +void SyncClock::ResetImpl() { + tab_ = 0; + tab_idx_ = 0; + size_ = 0; + blocks_ = 0; + release_store_tid_ = kInvalidTid; + release_store_reused_ = 0; + for (uptr i = 0; i < kDirtyTids; i++) + dirty_[i].tid = kInvalidTid; +} + void SyncClock::Resize(ClockCache *c, uptr nclk) { CPP_STAT_INC(StatClockReleaseResize); - if (RoundUpTo(nclk, ClockBlock::kClockCount) <= - RoundUpTo(size_, ClockBlock::kClockCount)) { - // Growing within the same block. + Unshare(c); + if (nclk <= capacity()) { // Memory is already allocated, just increase the size. size_ = nclk; return; } - if (nclk <= ClockBlock::kClockCount) { + if (size_ == 0) { // Grow from 0 to one-level table. CHECK_EQ(size_, 0); + CHECK_EQ(blocks_, 0); CHECK_EQ(tab_, 0); CHECK_EQ(tab_idx_, 0); - size_ = nclk; tab_idx_ = ctx->clock_alloc.Alloc(c); tab_ = ctx->clock_alloc.Map(tab_idx_); internal_memset(tab_, 0, sizeof(*tab_)); - return; - } - // Growing two-level table. - if (size_ == 0) { - // Allocate first level table. - tab_idx_ = ctx->clock_alloc.Alloc(c); - tab_ = ctx->clock_alloc.Map(tab_idx_); - internal_memset(tab_, 0, sizeof(*tab_)); - } else if (size_ <= ClockBlock::kClockCount) { - // Transform one-level table to two-level table. - u32 old = tab_idx_; - tab_idx_ = ctx->clock_alloc.Alloc(c); - tab_ = ctx->clock_alloc.Map(tab_idx_); - internal_memset(tab_, 0, sizeof(*tab_)); - tab_->table[0] = old; + atomic_store_relaxed(ref_ptr(tab_), 1); + size_ = 1; + } else if (size_ > blocks_ * ClockBlock::kClockCount) { + u32 idx = ctx->clock_alloc.Alloc(c); + ClockBlock *new_cb = ctx->clock_alloc.Map(idx); + uptr top = size_ - blocks_ * ClockBlock::kClockCount; + CHECK_LT(top, ClockBlock::kClockCount); + const uptr move = top * sizeof(tab_->clock[0]); + internal_memcpy(&new_cb->clock[0], tab_->clock, move); + internal_memset(&new_cb->clock[top], 0, sizeof(*new_cb) - move); + internal_memset(tab_->clock, 0, move); + append_block(idx); } - // At this point we have first level table allocated. + // At this point we have first level table allocated and all clock elements + // are evacuated from it to a second level block. // Add second level tables as necessary. - for (uptr i = RoundUpTo(size_, ClockBlock::kClockCount); - i < nclk; i += ClockBlock::kClockCount) { + while (nclk > capacity()) { u32 idx = ctx->clock_alloc.Alloc(c); ClockBlock *cb = ctx->clock_alloc.Map(idx); internal_memset(cb, 0, sizeof(*cb)); - CHECK_EQ(tab_->table[i/ClockBlock::kClockCount], 0); - tab_->table[i/ClockBlock::kClockCount] = idx; + append_block(idx); } size_ = nclk; } -// Sets a single element in the vector clock. -// This function is called only from weird places like AcquireGlobal. -void ThreadClock::set(unsigned tid, u64 v) { - DCHECK_LT(tid, kMaxTid); - DCHECK_GE(v, clk_[tid].epoch); - clk_[tid].epoch = v; - if (nclk_ <= tid) - nclk_ = tid + 1; - last_acquire_ = clk_[tid_].epoch; -} - -void ThreadClock::DebugDump(int(*printf)(const char *s, ...)) { - printf("clock=["); - for (uptr i = 0; i < nclk_; i++) - printf("%s%llu", i == 0 ? "" : ",", clk_[i].epoch); - printf("] reused=["); - for (uptr i = 0; i < nclk_; i++) - printf("%s%llu", i == 0 ? "" : ",", clk_[i].reused); - printf("] tid=%u/%u last_acq=%llu", - tid_, reused_, last_acquire_); +// Flushes all dirty elements into the main clock array. +void SyncClock::FlushDirty() { + for (unsigned i = 0; i < kDirtyTids; i++) { + Dirty *dirty = &dirty_[i]; + if (dirty->tid != kInvalidTid) { + CHECK_LT(dirty->tid, size_); + elem(dirty->tid).epoch = dirty->epoch; + dirty->tid = kInvalidTid; + } + } } -SyncClock::SyncClock() - : release_store_tid_(kInvalidTid) - , release_store_reused_() - , tab_() - , tab_idx_() - , size_() { - for (uptr i = 0; i < kDirtyTids; i++) - dirty_tids_[i] = kInvalidTid; +bool SyncClock::IsShared() const { + if (size_ == 0) + return false; + atomic_uint32_t *ref = ref_ptr(tab_); + u32 v = atomic_load(ref, memory_order_acquire); + CHECK_GT(v, 0); + return v > 1; } -SyncClock::~SyncClock() { - // Reset must be called before dtor. - CHECK_EQ(size_, 0); - CHECK_EQ(tab_, 0); - CHECK_EQ(tab_idx_, 0); +// Unshares the current clock if it's shared. +// Shared clocks are immutable, so they need to be unshared before any updates. +// Note: this does not apply to dirty entries as they are not shared. +void SyncClock::Unshare(ClockCache *c) { + if (!IsShared()) + return; + // First, copy current state into old. + SyncClock old; + old.tab_ = tab_; + old.tab_idx_ = tab_idx_; + old.size_ = size_; + old.blocks_ = blocks_; + old.release_store_tid_ = release_store_tid_; + old.release_store_reused_ = release_store_reused_; + for (unsigned i = 0; i < kDirtyTids; i++) + old.dirty_[i] = dirty_[i]; + // Then, clear current object. + ResetImpl(); + // Allocate brand new clock in the current object. + Resize(c, old.size_); + // Now copy state back into this object. + Iter old_iter(&old); + for (ClockElem &ce : *this) { + ce = *old_iter; + ++old_iter; + } + release_store_tid_ = old.release_store_tid_; + release_store_reused_ = old.release_store_reused_; + for (unsigned i = 0; i < kDirtyTids; i++) + dirty_[i] = old.dirty_[i]; + // Drop reference to old and delete if necessary. + old.Reset(c); } -void SyncClock::Reset(ClockCache *c) { - if (size_ == 0) { - // nothing - } else if (size_ <= ClockBlock::kClockCount) { - // One-level table. - ctx->clock_alloc.Free(c, tab_idx_); - } else { - // Two-level table. - for (uptr i = 0; i < size_; i += ClockBlock::kClockCount) - ctx->clock_alloc.Free(c, tab_->table[i / ClockBlock::kClockCount]); - ctx->clock_alloc.Free(c, tab_idx_); +// Can we cache this clock for future release operations? +ALWAYS_INLINE bool SyncClock::Cachable() const { + if (size_ == 0) + return false; + for (unsigned i = 0; i < kDirtyTids; i++) { + if (dirty_[i].tid != kInvalidTid) + return false; } - tab_ = 0; - tab_idx_ = 0; - size_ = 0; - release_store_tid_ = kInvalidTid; - release_store_reused_ = 0; - for (uptr i = 0; i < kDirtyTids; i++) - dirty_tids_[i] = kInvalidTid; + return atomic_load_relaxed(ref_ptr(tab_)) == 1; } -ClockElem &SyncClock::elem(unsigned tid) const { +// elem linearizes the two-level structure into linear array. +// Note: this is used only for one time accesses, vector operations use +// the iterator as it is much faster. +ALWAYS_INLINE ClockElem &SyncClock::elem(unsigned tid) const { DCHECK_LT(tid, size_); - if (size_ <= ClockBlock::kClockCount) + const uptr block = tid / ClockBlock::kClockCount; + DCHECK_LE(block, blocks_); + tid %= ClockBlock::kClockCount; + if (block == blocks_) return tab_->clock[tid]; - u32 idx = tab_->table[tid / ClockBlock::kClockCount]; + u32 idx = get_block(block); ClockBlock *cb = ctx->clock_alloc.Map(idx); - return cb->clock[tid % ClockBlock::kClockCount]; + return cb->clock[tid]; +} + +ALWAYS_INLINE uptr SyncClock::capacity() const { + if (size_ == 0) + return 0; + uptr ratio = sizeof(ClockBlock::clock[0]) / sizeof(ClockBlock::table[0]); + // How many clock elements we can fit into the first level block. + // +1 for ref counter. + uptr top = ClockBlock::kClockCount - RoundUpTo(blocks_ + 1, ratio) / ratio; + return blocks_ * ClockBlock::kClockCount + top; +} + +ALWAYS_INLINE u32 SyncClock::get_block(uptr bi) const { + DCHECK(size_); + DCHECK_LT(bi, blocks_); + return tab_->table[ClockBlock::kBlockIdx - bi]; +} + +ALWAYS_INLINE void SyncClock::append_block(u32 idx) { + uptr bi = blocks_++; + CHECK_EQ(get_block(bi), 0); + tab_->table[ClockBlock::kBlockIdx - bi] = idx; +} + +// Used only by tests. +u64 SyncClock::get(unsigned tid) const { + for (unsigned i = 0; i < kDirtyTids; i++) { + Dirty dirty = dirty_[i]; + if (dirty.tid == tid) + return dirty.epoch; + } + return elem(tid).epoch; +} + +// Used only by Iter test. +u64 SyncClock::get_clean(unsigned tid) const { + return elem(tid).epoch; } void SyncClock::DebugDump(int(*printf)(const char *s, ...)) { @@ -420,8 +567,32 @@ void SyncClock::DebugDump(int(*printf)(const char *s, ...)) { printf("] reused=["); for (uptr i = 0; i < size_; i++) printf("%s%llu", i == 0 ? "" : ",", elem(i).reused); - printf("] release_store_tid=%d/%d dirty_tids=%d/%d", + printf("] release_store_tid=%d/%d dirty_tids=%d[%llu]/%d[%llu]", release_store_tid_, release_store_reused_, - dirty_tids_[0], dirty_tids_[1]); + dirty_[0].tid, dirty_[0].epoch, + dirty_[1].tid, dirty_[1].epoch); +} + +void SyncClock::Iter::Next() { + // Finished with the current block, move on to the next one. + block_++; + if (block_ < parent_->blocks_) { + // Iterate over the next second level block. + u32 idx = parent_->get_block(block_); + ClockBlock *cb = ctx->clock_alloc.Map(idx); + pos_ = &cb->clock[0]; + end_ = pos_ + min(parent_->size_ - block_ * ClockBlock::kClockCount, + ClockBlock::kClockCount); + return; + } + if (block_ == parent_->blocks_ && + parent_->size_ > parent_->blocks_ * ClockBlock::kClockCount) { + // Iterate over elements in the first level block. + pos_ = &parent_->tab_->clock[0]; + end_ = pos_ + min(parent_->size_ - block_ * ClockBlock::kClockCount, + ClockBlock::kClockCount); + return; + } + parent_ = nullptr; // denotes end } } // namespace __tsan diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.h index 4e352cb..a891d7b 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_clock.h @@ -18,25 +18,6 @@ namespace __tsan { -struct ClockElem { - u64 epoch : kClkBits; - u64 reused : 64 - kClkBits; -}; - -struct ClockBlock { - static const uptr kSize = 512; - static const uptr kTableSize = kSize / sizeof(u32); - static const uptr kClockCount = kSize / sizeof(ClockElem); - - union { - u32 table[kTableSize]; - ClockElem clock[kClockCount]; - }; - - ClockBlock() { - } -}; - typedef DenseSlabAlloc<ClockBlock, 1<<16, 1<<10> ClockAlloc; typedef DenseSlabAllocCache ClockCache; @@ -46,84 +27,200 @@ class SyncClock { SyncClock(); ~SyncClock(); - uptr size() const { - return size_; - } + uptr size() const; - u64 get(unsigned tid) const { - return elem(tid).epoch; - } + // These are used only in tests. + u64 get(unsigned tid) const; + u64 get_clean(unsigned tid) const; void Resize(ClockCache *c, uptr nclk); void Reset(ClockCache *c); void DebugDump(int(*printf)(const char *s, ...)); + // Clock element iterator. + // Note: it iterates only over the table without regard to dirty entries. + class Iter { + public: + explicit Iter(SyncClock* parent); + Iter& operator++(); + bool operator!=(const Iter& other); + ClockElem &operator*(); + + private: + SyncClock *parent_; + // [pos_, end_) is the current continuous range of clock elements. + ClockElem *pos_; + ClockElem *end_; + int block_; // Current number of second level block. + + NOINLINE void Next(); + }; + + Iter begin(); + Iter end(); + private: - friend struct ThreadClock; + friend class ThreadClock; + friend class Iter; static const uptr kDirtyTids = 2; + struct Dirty { + u64 epoch : kClkBits; + u64 tid : 64 - kClkBits; // kInvalidId if not active + }; + unsigned release_store_tid_; unsigned release_store_reused_; - unsigned dirty_tids_[kDirtyTids]; - // tab_ contains indirect pointer to a 512b block using DenseSlabAlloc. - // If size_ <= 64, then tab_ points to an array with 64 ClockElem's. - // Otherwise, tab_ points to an array with 128 u32 elements, + Dirty dirty_[kDirtyTids]; + // If size_ is 0, tab_ is nullptr. + // If size <= 64 (kClockCount), tab_ contains pointer to an array with + // 64 ClockElem's (ClockBlock::clock). + // Otherwise, tab_ points to an array with up to 127 u32 elements, // each pointing to the second-level 512b block with 64 ClockElem's. + // Unused space in the first level ClockBlock is used to store additional + // clock elements. + // The last u32 element in the first level ClockBlock is always used as + // reference counter. + // + // See the following scheme for details. + // All memory blocks are 512 bytes (allocated from ClockAlloc). + // Clock (clk) elements are 64 bits. + // Idx and ref are 32 bits. + // + // tab_ + // | + // \/ + // +----------------------------------------------------+ + // | clk128 | clk129 | ...unused... | idx1 | idx0 | ref | + // +----------------------------------------------------+ + // | | + // | \/ + // | +----------------+ + // | | clk0 ... clk63 | + // | +----------------+ + // \/ + // +------------------+ + // | clk64 ... clk127 | + // +------------------+ + // + // Note: dirty entries, if active, always override what's stored in the clock. ClockBlock *tab_; u32 tab_idx_; - u32 size_; - + u16 size_; + u16 blocks_; // Number of second level blocks. + + void Unshare(ClockCache *c); + bool IsShared() const; + bool Cachable() const; + void ResetImpl(); + void FlushDirty(); + uptr capacity() const; + u32 get_block(uptr bi) const; + void append_block(u32 idx); ClockElem &elem(unsigned tid) const; }; // The clock that lives in threads. -struct ThreadClock { +class ThreadClock { public: typedef DenseSlabAllocCache Cache; explicit ThreadClock(unsigned tid, unsigned reused = 0); - u64 get(unsigned tid) const { - DCHECK_LT(tid, kMaxTidInClock); - return clk_[tid].epoch; - } - - void set(unsigned tid, u64 v); - - void set(u64 v) { - DCHECK_GE(v, clk_[tid_].epoch); - clk_[tid_].epoch = v; - } + u64 get(unsigned tid) const; + void set(ClockCache *c, unsigned tid, u64 v); + void set(u64 v); + void tick(); + uptr size() const; - void tick() { - clk_[tid_].epoch++; - } - - uptr size() const { - return nclk_; - } - - void acquire(ClockCache *c, const SyncClock *src); - void release(ClockCache *c, SyncClock *dst) const; + void acquire(ClockCache *c, SyncClock *src); + void release(ClockCache *c, SyncClock *dst); void acq_rel(ClockCache *c, SyncClock *dst); - void ReleaseStore(ClockCache *c, SyncClock *dst) const; + void ReleaseStore(ClockCache *c, SyncClock *dst); + void ResetCached(ClockCache *c); void DebugReset(); void DebugDump(int(*printf)(const char *s, ...)); private: static const uptr kDirtyTids = SyncClock::kDirtyTids; + // Index of the thread associated with he clock ("current thread"). const unsigned tid_; - const unsigned reused_; + const unsigned reused_; // tid_ reuse count. + // Current thread time when it acquired something from other threads. u64 last_acquire_; + + // Cached SyncClock (without dirty entries and release_store_tid_). + // We reuse it for subsequent store-release operations without intervening + // acquire operations. Since it is shared (and thus constant), clock value + // for the current thread is then stored in dirty entries in the SyncClock. + // We host a refernece to the table while it is cached here. + u32 cached_idx_; + u16 cached_size_; + u16 cached_blocks_; + + // Number of active elements in the clk_ table (the rest is zeros). uptr nclk_; - ClockElem clk_[kMaxTidInClock]; + u64 clk_[kMaxTidInClock]; // Fixed size vector clock. bool IsAlreadyAcquired(const SyncClock *src) const; - void UpdateCurrentThread(SyncClock *dst) const; + void UpdateCurrentThread(ClockCache *c, SyncClock *dst) const; }; +ALWAYS_INLINE u64 ThreadClock::get(unsigned tid) const { + DCHECK_LT(tid, kMaxTidInClock); + return clk_[tid]; +} + +ALWAYS_INLINE void ThreadClock::set(u64 v) { + DCHECK_GE(v, clk_[tid_]); + clk_[tid_] = v; +} + +ALWAYS_INLINE void ThreadClock::tick() { + clk_[tid_]++; +} + +ALWAYS_INLINE uptr ThreadClock::size() const { + return nclk_; +} + +ALWAYS_INLINE SyncClock::Iter SyncClock::begin() { + return Iter(this); +} + +ALWAYS_INLINE SyncClock::Iter SyncClock::end() { + return Iter(nullptr); +} + +ALWAYS_INLINE uptr SyncClock::size() const { + return size_; +} + +ALWAYS_INLINE SyncClock::Iter::Iter(SyncClock* parent) + : parent_(parent) + , pos_(nullptr) + , end_(nullptr) + , block_(-1) { + if (parent) + Next(); +} + +ALWAYS_INLINE SyncClock::Iter& SyncClock::Iter::operator++() { + pos_++; + if (UNLIKELY(pos_ >= end_)) + Next(); + return *this; +} + +ALWAYS_INLINE bool SyncClock::Iter::operator!=(const SyncClock::Iter& other) { + return parent_ != other.parent_; +} + +ALWAYS_INLINE ClockElem &SyncClock::Iter::operator*() { + return *pos_; +} } // namespace __tsan #endif // TSAN_CLOCK_H diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_debugging.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_debugging.cc index d9fb686..a44b136 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_debugging.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_debugging.cc @@ -24,6 +24,7 @@ static const char *ReportTypeDescription(ReportType typ) { if (typ == ReportTypeVptrRace) return "data-race-vptr"; if (typ == ReportTypeUseAfterFree) return "heap-use-after-free"; if (typ == ReportTypeVptrUseAfterFree) return "heap-use-after-free-vptr"; + if (typ == ReportTypeExternalRace) return "external-race"; if (typ == ReportTypeThreadLeak) return "thread-leak"; if (typ == ReportTypeMutexDestroyLocked) return "locked-mutex-destroy"; if (typ == ReportTypeMutexDoubleLock) return "mutex-double-lock"; @@ -127,6 +128,16 @@ int __tsan_get_report_loc(void *report, uptr idx, const char **type, } SANITIZER_INTERFACE_ATTRIBUTE +int __tsan_get_report_loc_object_type(void *report, uptr idx, + const char **object_type) { + const ReportDesc *rep = (ReportDesc *)report; + CHECK_LT(idx, rep->locs.Size()); + ReportLocation *loc = rep->locs[idx]; + *object_type = GetObjectTypeFromTag(loc->external_tag); + return 1; +} + +SANITIZER_INTERFACE_ATTRIBUTE int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr, int *destroyed, void **trace, uptr trace_size) { const ReportDesc *rep = (ReportDesc *)report; @@ -140,7 +151,7 @@ int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr, } SANITIZER_INTERFACE_ATTRIBUTE -int __tsan_get_report_thread(void *report, uptr idx, int *tid, uptr *os_id, +int __tsan_get_report_thread(void *report, uptr idx, int *tid, tid_t *os_id, int *running, const char **name, int *parent_tid, void **trace, uptr trace_size) { const ReportDesc *rep = (ReportDesc *)report; @@ -217,7 +228,7 @@ const char *__tsan_locate_address(uptr addr, char *name, uptr name_size, SANITIZER_INTERFACE_ATTRIBUTE int __tsan_get_alloc_stack(uptr addr, uptr *trace, uptr size, int *thread_id, - uptr *os_id) { + tid_t *os_id) { MBlock *b = 0; Allocator *a = allocator(); if (a->PointerIsMine((void *)addr)) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_defs.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_defs.h index 55580a5..3c775de 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_defs.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_defs.h @@ -38,15 +38,40 @@ namespace __tsan { +const int kClkBits = 42; +const unsigned kMaxTidReuse = (1 << (64 - kClkBits)) - 1; + +struct ClockElem { + u64 epoch : kClkBits; + u64 reused : 64 - kClkBits; // tid reuse count +}; + +struct ClockBlock { + static const uptr kSize = 512; + static const uptr kTableSize = kSize / sizeof(u32); + static const uptr kClockCount = kSize / sizeof(ClockElem); + static const uptr kRefIdx = kTableSize - 1; + static const uptr kBlockIdx = kTableSize - 2; + + union { + u32 table[kTableSize]; + ClockElem clock[kClockCount]; + }; + + ClockBlock() { + } +}; + const int kTidBits = 13; -const unsigned kMaxTid = 1 << kTidBits; +// Reduce kMaxTid by kClockCount because one slot in ClockBlock table is +// occupied by reference counter, so total number of elements we can store +// in SyncClock is kClockCount * (kTableSize - 1). +const unsigned kMaxTid = (1 << kTidBits) - ClockBlock::kClockCount; #if !SANITIZER_GO const unsigned kMaxTidInClock = kMaxTid * 2; // This includes msb 'freed' bit. #else const unsigned kMaxTidInClock = kMaxTid; // Go does not track freed memory. #endif -const int kClkBits = 42; -const unsigned kMaxTidReuse = (1 << (64 - kClkBits)) - 1; const uptr kShadowStackSize = 64 * 1024; // Count of shadow values in a shadow cell. @@ -74,7 +99,7 @@ const bool kCollectHistory = false; const bool kCollectHistory = true; #endif -const unsigned kInvalidTid = (unsigned)-1; +const u16 kInvalidTid = kMaxTid + 1; // The following "build consistency" machinery ensures that all source files // are built in the same configuration. Inconsistent builds lead to @@ -149,13 +174,23 @@ class RegionAlloc; // Descriptor of user's memory block. struct MBlock { - u64 siz; + u64 siz : 48; + u64 tag : 16; u32 stk; u16 tid; }; COMPILER_CHECK(sizeof(MBlock) == 16); +enum ExternalTag : uptr { + kExternalTagNone = 0, + kExternalTagSwiftModifyingAccess = 1, + kExternalTagFirstUserAvailable = 2, + kExternalTagMax = 1024, + // Don't set kExternalTagMax over 65,536, since MBlock only stores tags + // as 16-bit values, see tsan_defs.h. +}; + } // namespace __tsan #endif // TSAN_DEFS_H diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_dense_alloc.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_dense_alloc.h index e9815c9..16dbdf3 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_dense_alloc.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_dense_alloc.h @@ -39,7 +39,7 @@ class DenseSlabAlloc { typedef DenseSlabAllocCache Cache; typedef typename Cache::IndexT IndexT; - DenseSlabAlloc() { + explicit DenseSlabAlloc(const char *name) { // Check that kL1Size and kL2Size are sane. CHECK_EQ(kL1Size & (kL1Size - 1), 0); CHECK_EQ(kL2Size & (kL2Size - 1), 0); @@ -49,6 +49,7 @@ class DenseSlabAlloc { internal_memset(map_, 0, sizeof(map_)); freelist_ = 0; fillpos_ = 0; + name_ = name; } ~DenseSlabAlloc() { @@ -96,15 +97,19 @@ class DenseSlabAlloc { SpinMutex mtx_; IndexT freelist_; uptr fillpos_; + const char *name_; void Refill(Cache *c) { SpinMutexLock lock(&mtx_); if (freelist_ == 0) { if (fillpos_ == kL1Size) { - Printf("ThreadSanitizer: DenseSlabAllocator overflow. Dying.\n"); + Printf("ThreadSanitizer: %s overflow (%zu*%zu). Dying.\n", + name_, kL1Size, kL2Size); Die(); } - T *batch = (T*)MmapOrDie(kL2Size * sizeof(T), "DenseSlabAllocator"); + VPrintf(2, "ThreadSanitizer: growing %s: %zu out of %zu*%zu\n", + name_, fillpos_, kL1Size, kL2Size); + T *batch = (T*)MmapOrDie(kL2Size * sizeof(T), name_); // Reserve 0 as invalid index. IndexT start = fillpos_ == 0 ? 1 : 0; for (IndexT i = start; i < kL2Size; i++) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_external.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_external.cc new file mode 100644 index 0000000..6c0e947 --- /dev/null +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_external.cc @@ -0,0 +1,125 @@ +//===-- tsan_external.cc --------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of ThreadSanitizer (TSan), a race detector. +// +//===----------------------------------------------------------------------===// +#include "tsan_rtl.h" +#include "tsan_interceptors.h" + +namespace __tsan { + +#define CALLERPC ((uptr)__builtin_return_address(0)) + +struct TagData { + const char *object_type; + const char *header; +}; + +static TagData registered_tags[kExternalTagMax] = { + {}, + {"Swift variable", "Swift access race"}, +}; +static atomic_uint32_t used_tags{kExternalTagFirstUserAvailable}; // NOLINT. +static TagData *GetTagData(uptr tag) { + // Invalid/corrupted tag? Better return NULL and let the caller deal with it. + if (tag >= atomic_load(&used_tags, memory_order_relaxed)) return nullptr; + return ®istered_tags[tag]; +} + +const char *GetObjectTypeFromTag(uptr tag) { + TagData *tag_data = GetTagData(tag); + return tag_data ? tag_data->object_type : nullptr; +} + +const char *GetReportHeaderFromTag(uptr tag) { + TagData *tag_data = GetTagData(tag); + return tag_data ? tag_data->header : nullptr; +} + +void InsertShadowStackFrameForTag(ThreadState *thr, uptr tag) { + FuncEntry(thr, (uptr)®istered_tags[tag]); +} + +uptr TagFromShadowStackFrame(uptr pc) { + uptr tag_count = atomic_load(&used_tags, memory_order_relaxed); + void *pc_ptr = (void *)pc; + if (pc_ptr < GetTagData(0) || pc_ptr > GetTagData(tag_count - 1)) + return 0; + return (TagData *)pc_ptr - GetTagData(0); +} + +#if !SANITIZER_GO + +typedef void(*AccessFunc)(ThreadState *, uptr, uptr, int); +void ExternalAccess(void *addr, void *caller_pc, void *tag, AccessFunc access) { + CHECK_LT(tag, atomic_load(&used_tags, memory_order_relaxed)); + ThreadState *thr = cur_thread(); + if (caller_pc) FuncEntry(thr, (uptr)caller_pc); + InsertShadowStackFrameForTag(thr, (uptr)tag); + bool in_ignored_lib; + if (!caller_pc || !libignore()->IsIgnored((uptr)caller_pc, &in_ignored_lib)) { + access(thr, CALLERPC, (uptr)addr, kSizeLog1); + } + FuncExit(thr); + if (caller_pc) FuncExit(thr); +} + +extern "C" { +SANITIZER_INTERFACE_ATTRIBUTE +void *__tsan_external_register_tag(const char *object_type) { + uptr new_tag = atomic_fetch_add(&used_tags, 1, memory_order_relaxed); + CHECK_LT(new_tag, kExternalTagMax); + GetTagData(new_tag)->object_type = internal_strdup(object_type); + char header[127] = {0}; + internal_snprintf(header, sizeof(header), "race on %s", object_type); + GetTagData(new_tag)->header = internal_strdup(header); + return (void *)new_tag; +} + +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_register_header(void *tag, const char *header) { + CHECK_GE((uptr)tag, kExternalTagFirstUserAvailable); + CHECK_LT((uptr)tag, kExternalTagMax); + atomic_uintptr_t *header_ptr = + (atomic_uintptr_t *)&GetTagData((uptr)tag)->header; + header = internal_strdup(header); + char *old_header = + (char *)atomic_exchange(header_ptr, (uptr)header, memory_order_seq_cst); + if (old_header) internal_free(old_header); +} + +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_assign_tag(void *addr, void *tag) { + CHECK_LT(tag, atomic_load(&used_tags, memory_order_relaxed)); + Allocator *a = allocator(); + MBlock *b = nullptr; + if (a->PointerIsMine((void *)addr)) { + void *block_begin = a->GetBlockBegin((void *)addr); + if (block_begin) b = ctx->metamap.GetBlock((uptr)block_begin); + } + if (b) { + b->tag = (uptr)tag; + } +} + +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_read(void *addr, void *caller_pc, void *tag) { + ExternalAccess(addr, caller_pc, tag, MemoryRead); +} + +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_write(void *addr, void *caller_pc, void *tag) { + ExternalAccess(addr, caller_pc, tag, MemoryWrite); +} +} // extern "C" + +#endif // !SANITIZER_GO + +} // namespace __tsan diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.cc index d8d4746..89e22a1 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.cc @@ -21,10 +21,6 @@ namespace __tsan { -Flags *flags() { - return &ctx->flags; -} - // Can be overriden in frontend. #ifdef TSAN_EXTERNAL_HOOKS extern "C" const char* __tsan_default_options(); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.h index e2f6b3c..66740de 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.h @@ -28,7 +28,6 @@ struct Flags : DDFlags { void ParseFromString(const char *str); }; -Flags *flags(); void InitializeFlags(Flags *flags, const char *env); } // namespace __tsan diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.inc b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.inc index a48545c..e9b3e35 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.inc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_flags.inc @@ -79,7 +79,7 @@ TSAN_FLAG(bool, die_after_fork, true, TSAN_FLAG(const char *, suppressions, "", "Suppressions file name.") TSAN_FLAG(bool, ignore_interceptors_accesses, false, "Ignore reads and writes from all interceptors.") -TSAN_FLAG(bool, ignore_noninstrumented_modules, false, +TSAN_FLAG(bool, ignore_noninstrumented_modules, SANITIZER_MAC ? true : false, "Interceptors should only detect races when called from instrumented " "modules.") TSAN_FLAG(bool, shared_ptr_interceptor, true, diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc index 898f32d..001123f 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc @@ -14,10 +14,12 @@ //===----------------------------------------------------------------------===// #include "sanitizer_common/sanitizer_atomic.h" +#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_libc.h" #include "sanitizer_common/sanitizer_linux.h" #include "sanitizer_common/sanitizer_platform_limits_posix.h" #include "sanitizer_common/sanitizer_placement_new.h" +#include "sanitizer_common/sanitizer_posix.h" #include "sanitizer_common/sanitizer_stacktrace.h" #include "sanitizer_common/sanitizer_tls_get_addr.h" #include "interception/interception.h" @@ -29,30 +31,18 @@ #include "tsan_mman.h" #include "tsan_fd.h" -#if SANITIZER_POSIX -#include "sanitizer_common/sanitizer_posix.h" -#endif using namespace __tsan; // NOLINT #if SANITIZER_FREEBSD || SANITIZER_MAC -#define __errno_location __error #define stdout __stdoutp #define stderr __stderrp #endif #if SANITIZER_ANDROID -#define __errno_location __errno #define mallopt(a, b) #endif -#if SANITIZER_LINUX || SANITIZER_FREEBSD -#define PTHREAD_CREATE_DETACHED 1 -#elif SANITIZER_MAC -#define PTHREAD_CREATE_DETACHED 2 -#endif - - #ifdef __mips__ const int kSigCount = 129; #else @@ -93,7 +83,6 @@ DECLARE_REAL_AND_INTERCEPTOR(void *, malloc, uptr size) DECLARE_REAL_AND_INTERCEPTOR(void, free, void *ptr) extern "C" void *pthread_self(); extern "C" void _exit(int status); -extern "C" int *__errno_location(); extern "C" int fileno_unlocked(void *stream); extern "C" int dirfd(void *dirp); #if !SANITIZER_FREEBSD && !SANITIZER_ANDROID @@ -107,9 +96,6 @@ const int PTHREAD_MUTEX_RECURSIVE_NP = 1; const int PTHREAD_MUTEX_RECURSIVE = 2; const int PTHREAD_MUTEX_RECURSIVE_NP = 2; #endif -const int EINVAL = 22; -const int EBUSY = 16; -const int EOWNERDEAD = 130; #if !SANITIZER_FREEBSD && !SANITIZER_MAC const int EPOLL_CTL_ADD = 1; #endif @@ -139,8 +125,6 @@ typedef long long_t; // NOLINT # define F_TLOCK 2 /* Test and lock a region for exclusive use. */ # define F_TEST 3 /* Test a region for other processes locks. */ -#define errno (*__errno_location()) - typedef void (*sighandler_t)(int sig); typedef void (*sigactionhandler_t)(int sig, my_siginfo_t *siginfo, void *uctx); @@ -219,7 +203,7 @@ struct ThreadSignalContext { // The object is 64-byte aligned, because we want hot data to be located in // a single cache line if possible (it's accessed in every interceptor). static ALIGNED(64) char libignore_placeholder[sizeof(LibIgnore)]; -static LibIgnore *libignore() { +LibIgnore *libignore() { return reinterpret_cast<LibIgnore*>(&libignore_placeholder[0]); } @@ -277,7 +261,8 @@ ScopedInterceptor::~ScopedInterceptor() { void ScopedInterceptor::EnableIgnores() { if (ignoring_) { - ThreadIgnoreBegin(thr_, pc_); + ThreadIgnoreBegin(thr_, pc_, /*save_stack=*/false); + if (flags()->ignore_noninstrumented_modules) thr_->suppress_reports++; if (in_ignored_lib_) { DCHECK(!thr_->in_ignored_lib); thr_->in_ignored_lib = true; @@ -288,6 +273,7 @@ void ScopedInterceptor::EnableIgnores() { void ScopedInterceptor::DisableIgnores() { if (ignoring_) { ThreadIgnoreEnd(thr_, pc_); + if (flags()->ignore_noninstrumented_modules) thr_->suppress_reports--; if (in_ignored_lib_) { DCHECK(thr_->in_ignored_lib); thr_->in_ignored_lib = false; @@ -473,8 +459,14 @@ static void SetJmp(ThreadState *thr, uptr sp, uptr mangled_sp) { static void LongJmp(ThreadState *thr, uptr *env) { #ifdef __powerpc__ uptr mangled_sp = env[0]; -#elif SANITIZER_FREEBSD || SANITIZER_MAC +#elif SANITIZER_FREEBSD uptr mangled_sp = env[2]; +#elif SANITIZER_MAC +# ifdef __aarch64__ + uptr mangled_sp = env[13]; +# else + uptr mangled_sp = env[2]; +# endif #elif defined(SANITIZER_LINUX) # ifdef __aarch64__ uptr mangled_sp = env[13]; @@ -672,7 +664,7 @@ static bool fix_mmap_addr(void **addr, long_t sz, int flags) { if (*addr) { if (!IsAppMem((uptr)*addr) || !IsAppMem((uptr)*addr + sz - 1)) { if (flags & MAP_FIXED) { - errno = EINVAL; + errno = errno_EINVAL; return false; } else { *addr = 0; @@ -881,7 +873,7 @@ extern "C" void *__tsan_thread_start_func(void *arg) { internal_sched_yield(); Processor *proc = ProcCreate(); ProcWire(proc, thr); - ThreadStart(thr, tid, GetTid()); + ThreadStart(thr, tid, GetTid(), /*workerthread*/ false); atomic_store(&p->tid, 0, memory_order_release); } void *res = callback(param); @@ -928,8 +920,7 @@ TSAN_INTERCEPTOR(int, pthread_create, ThreadIgnoreEnd(thr, pc); } if (res == 0) { - int tid = ThreadCreate(thr, pc, *(uptr*)th, - detached == PTHREAD_CREATE_DETACHED); + int tid = ThreadCreate(thr, pc, *(uptr*)th, IsStateDetached(detached)); CHECK_NE(tid, 0); // Synchronization on p.tid serves two purposes: // 1. ThreadCreate must finish before the new thread starts. @@ -1025,7 +1016,7 @@ static void cond_mutex_unlock(CondMutexUnlockCtx *arg) { ThreadSignalContext *ctx = SigCtx(arg->thr); CHECK_EQ(atomic_load(&ctx->in_blocking_func, memory_order_relaxed), 1); atomic_store(&ctx->in_blocking_func, 0, memory_order_relaxed); - MutexLock(arg->thr, arg->pc, (uptr)arg->m); + MutexPostLock(arg->thr, arg->pc, (uptr)arg->m, MutexFlagDoPreLockOnPostLock); // Undo BlockingCall ctor effects. arg->thr->ignore_interceptors--; arg->si->~ScopedInterceptor(); @@ -1054,7 +1045,7 @@ static int cond_wait(ThreadState *thr, uptr pc, ScopedInterceptor *si, fn, c, m, t, (void (*)(void *arg))cond_mutex_unlock, &arg); } if (res == errno_EOWNERDEAD) MutexRepair(thr, pc, (uptr)m); - MutexLock(thr, pc, (uptr)m); + MutexPostLock(thr, pc, (uptr)m, MutexFlagDoPreLockOnPostLock); return res; } @@ -1114,14 +1105,15 @@ TSAN_INTERCEPTOR(int, pthread_mutex_init, void *m, void *a) { SCOPED_TSAN_INTERCEPTOR(pthread_mutex_init, m, a); int res = REAL(pthread_mutex_init)(m, a); if (res == 0) { - bool recursive = false; + u32 flagz = 0; if (a) { int type = 0; if (REAL(pthread_mutexattr_gettype)(a, &type) == 0) - recursive = (type == PTHREAD_MUTEX_RECURSIVE - || type == PTHREAD_MUTEX_RECURSIVE_NP); + if (type == PTHREAD_MUTEX_RECURSIVE || + type == PTHREAD_MUTEX_RECURSIVE_NP) + flagz |= MutexFlagWriteReentrant; } - MutexCreate(thr, pc, (uptr)m, false, recursive, false); + MutexCreate(thr, pc, (uptr)m, flagz); } return res; } @@ -1129,7 +1121,7 @@ TSAN_INTERCEPTOR(int, pthread_mutex_init, void *m, void *a) { TSAN_INTERCEPTOR(int, pthread_mutex_destroy, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_mutex_destroy, m); int res = REAL(pthread_mutex_destroy)(m); - if (res == 0 || res == EBUSY) { + if (res == 0 || res == errno_EBUSY) { MutexDestroy(thr, pc, (uptr)m); } return res; @@ -1138,10 +1130,10 @@ TSAN_INTERCEPTOR(int, pthread_mutex_destroy, void *m) { TSAN_INTERCEPTOR(int, pthread_mutex_trylock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_mutex_trylock, m); int res = REAL(pthread_mutex_trylock)(m); - if (res == EOWNERDEAD) + if (res == errno_EOWNERDEAD) MutexRepair(thr, pc, (uptr)m); - if (res == 0 || res == EOWNERDEAD) - MutexLock(thr, pc, (uptr)m, /*rec=*/1, /*try_lock=*/true); + if (res == 0 || res == errno_EOWNERDEAD) + MutexPostLock(thr, pc, (uptr)m, MutexFlagTryLock); return res; } @@ -1150,7 +1142,7 @@ TSAN_INTERCEPTOR(int, pthread_mutex_timedlock, void *m, void *abstime) { SCOPED_TSAN_INTERCEPTOR(pthread_mutex_timedlock, m, abstime); int res = REAL(pthread_mutex_timedlock)(m, abstime); if (res == 0) { - MutexLock(thr, pc, (uptr)m); + MutexPostLock(thr, pc, (uptr)m, MutexFlagTryLock); } return res; } @@ -1161,7 +1153,7 @@ TSAN_INTERCEPTOR(int, pthread_spin_init, void *m, int pshared) { SCOPED_TSAN_INTERCEPTOR(pthread_spin_init, m, pshared); int res = REAL(pthread_spin_init)(m, pshared); if (res == 0) { - MutexCreate(thr, pc, (uptr)m, false, false, false); + MutexCreate(thr, pc, (uptr)m); } return res; } @@ -1177,9 +1169,10 @@ TSAN_INTERCEPTOR(int, pthread_spin_destroy, void *m) { TSAN_INTERCEPTOR(int, pthread_spin_lock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_spin_lock, m); + MutexPreLock(thr, pc, (uptr)m); int res = REAL(pthread_spin_lock)(m); if (res == 0) { - MutexLock(thr, pc, (uptr)m); + MutexPostLock(thr, pc, (uptr)m); } return res; } @@ -1188,7 +1181,7 @@ TSAN_INTERCEPTOR(int, pthread_spin_trylock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_spin_trylock, m); int res = REAL(pthread_spin_trylock)(m); if (res == 0) { - MutexLock(thr, pc, (uptr)m, /*rec=*/1, /*try_lock=*/true); + MutexPostLock(thr, pc, (uptr)m, MutexFlagTryLock); } return res; } @@ -1205,7 +1198,7 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_init, void *m, void *a) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_init, m, a); int res = REAL(pthread_rwlock_init)(m, a); if (res == 0) { - MutexCreate(thr, pc, (uptr)m, true, false, false); + MutexCreate(thr, pc, (uptr)m); } return res; } @@ -1221,9 +1214,10 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_destroy, void *m) { TSAN_INTERCEPTOR(int, pthread_rwlock_rdlock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_rdlock, m); + MutexPreReadLock(thr, pc, (uptr)m); int res = REAL(pthread_rwlock_rdlock)(m); if (res == 0) { - MutexReadLock(thr, pc, (uptr)m); + MutexPostReadLock(thr, pc, (uptr)m); } return res; } @@ -1232,7 +1226,7 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_tryrdlock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_tryrdlock, m); int res = REAL(pthread_rwlock_tryrdlock)(m); if (res == 0) { - MutexReadLock(thr, pc, (uptr)m, /*try_lock=*/true); + MutexPostReadLock(thr, pc, (uptr)m, MutexFlagTryLock); } return res; } @@ -1242,7 +1236,7 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_timedrdlock, void *m, void *abstime) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_timedrdlock, m, abstime); int res = REAL(pthread_rwlock_timedrdlock)(m, abstime); if (res == 0) { - MutexReadLock(thr, pc, (uptr)m); + MutexPostReadLock(thr, pc, (uptr)m); } return res; } @@ -1250,9 +1244,10 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_timedrdlock, void *m, void *abstime) { TSAN_INTERCEPTOR(int, pthread_rwlock_wrlock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_wrlock, m); + MutexPreLock(thr, pc, (uptr)m); int res = REAL(pthread_rwlock_wrlock)(m); if (res == 0) { - MutexLock(thr, pc, (uptr)m); + MutexPostLock(thr, pc, (uptr)m); } return res; } @@ -1261,7 +1256,7 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_trywrlock, void *m) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_trywrlock, m); int res = REAL(pthread_rwlock_trywrlock)(m); if (res == 0) { - MutexLock(thr, pc, (uptr)m, /*rec=*/1, /*try_lock=*/true); + MutexPostLock(thr, pc, (uptr)m, MutexFlagTryLock); } return res; } @@ -1271,7 +1266,7 @@ TSAN_INTERCEPTOR(int, pthread_rwlock_timedwrlock, void *m, void *abstime) { SCOPED_TSAN_INTERCEPTOR(pthread_rwlock_timedwrlock, m, abstime); int res = REAL(pthread_rwlock_timedwrlock)(m, abstime); if (res == 0) { - MutexLock(thr, pc, (uptr)m); + MutexPostLock(thr, pc, (uptr)m, MutexFlagTryLock); } return res; } @@ -1315,7 +1310,7 @@ TSAN_INTERCEPTOR(int, pthread_barrier_wait, void *b) { TSAN_INTERCEPTOR(int, pthread_once, void *o, void (*f)()) { SCOPED_INTERCEPTOR_RAW(pthread_once, o, f); if (o == 0 || f == 0) - return EINVAL; + return errno_EINVAL; atomic_uint32_t *a; if (!SANITIZER_MAC) a = static_cast<atomic_uint32_t*>(o); @@ -1644,24 +1639,6 @@ TSAN_INTERCEPTOR(void*, tmpfile64, int fake) { #define TSAN_MAYBE_INTERCEPT_TMPFILE64 #endif -TSAN_INTERCEPTOR(uptr, fread, void *ptr, uptr size, uptr nmemb, void *f) { - // libc file streams can call user-supplied functions, see fopencookie. - { - SCOPED_TSAN_INTERCEPTOR(fread, ptr, size, nmemb, f); - MemoryAccessRange(thr, pc, (uptr)ptr, size * nmemb, true); - } - return REAL(fread)(ptr, size, nmemb, f); -} - -TSAN_INTERCEPTOR(uptr, fwrite, const void *p, uptr size, uptr nmemb, void *f) { - // libc file streams can call user-supplied functions, see fopencookie. - { - SCOPED_TSAN_INTERCEPTOR(fwrite, p, size, nmemb, f); - MemoryAccessRange(thr, pc, (uptr)p, size * nmemb, false); - } - return REAL(fwrite)(p, size, nmemb, f); -} - static void FlushStreams() { // Flushing all the streams here may freeze the process if a child thread is // performing file stream operations at the same time. @@ -2251,8 +2228,12 @@ static void HandleRecvmsg(ThreadState *thr, uptr pc, #define COMMON_INTERCEPTOR_ON_EXIT(ctx) \ OnExit(((TsanInterceptorContext *) ctx)->thr) -#define COMMON_INTERCEPTOR_MUTEX_LOCK(ctx, m) \ - MutexLock(((TsanInterceptorContext *)ctx)->thr, \ +#define COMMON_INTERCEPTOR_MUTEX_PRE_LOCK(ctx, m) \ + MutexPreLock(((TsanInterceptorContext *)ctx)->thr, \ + ((TsanInterceptorContext *)ctx)->pc, (uptr)m) + +#define COMMON_INTERCEPTOR_MUTEX_POST_LOCK(ctx, m) \ + MutexPostLock(((TsanInterceptorContext *)ctx)->thr, \ ((TsanInterceptorContext *)ctx)->pc, (uptr)m) #define COMMON_INTERCEPTOR_MUTEX_UNLOCK(ctx, m) \ diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.h index 72534f4..de47466 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.h @@ -19,6 +19,8 @@ class ScopedInterceptor { bool ignoring_; }; +LibIgnore *libignore(); + } // namespace __tsan #define SCOPED_INTERCEPTOR_RAW(func, ...) \ diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cc index fc5eb04..4f10794 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cc @@ -21,7 +21,10 @@ #include "tsan_interface_ann.h" #include <libkern/OSAtomic.h> + +#if defined(__has_include) && __has_include(<xpc/xpc.h>) #include <xpc/xpc.h> +#endif // #if defined(__has_include) && __has_include(<xpc/xpc.h>) typedef long long_t; // NOLINT @@ -235,6 +238,8 @@ TSAN_INTERCEPTOR(void, os_lock_unlock, void *lock) { REAL(os_lock_unlock)(lock); } +#if defined(__has_include) && __has_include(<xpc/xpc.h>) + TSAN_INTERCEPTOR(void, xpc_connection_set_event_handler, xpc_connection_t connection, xpc_handler_t handler) { SCOPED_TSAN_INTERCEPTOR(xpc_connection_set_event_handler, connection, @@ -281,6 +286,14 @@ TSAN_INTERCEPTOR(void, xpc_connection_send_message_with_reply, (connection, message, replyq, new_handler); } +TSAN_INTERCEPTOR(void, xpc_connection_cancel, xpc_connection_t connection) { + SCOPED_TSAN_INTERCEPTOR(xpc_connection_cancel, connection); + Release(thr, pc, (uptr)connection); + REAL(xpc_connection_cancel)(connection); +} + +#endif // #if defined(__has_include) && __has_include(<xpc/xpc.h>) + // On macOS, libc++ is always linked dynamically, so intercepting works the // usual way. #define STDCXX_INTERCEPTOR TSAN_INTERCEPTOR diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface.h index 4e342a5..a80a489 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface.h @@ -18,6 +18,7 @@ #include <sanitizer_common/sanitizer_internal_defs.h> using __sanitizer::uptr; +using __sanitizer::tid_t; // This header should NOT include any other headers. // All functions in this header are extern "C" and start with __tsan_. @@ -79,6 +80,17 @@ SANITIZER_INTERFACE_ATTRIBUTE void __tsan_ignore_thread_begin(); SANITIZER_INTERFACE_ATTRIBUTE void __tsan_ignore_thread_end(); SANITIZER_INTERFACE_ATTRIBUTE +void *__tsan_external_register_tag(const char *object_type); +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_register_header(void *tag, const char *header); +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_assign_tag(void *addr, void *tag); +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_read(void *addr, void *caller_pc, void *tag); +SANITIZER_INTERFACE_ATTRIBUTE +void __tsan_external_write(void *addr, void *caller_pc, void *tag); + +SANITIZER_INTERFACE_ATTRIBUTE void __tsan_read_range(void *addr, unsigned long size); // NOLINT SANITIZER_INTERFACE_ATTRIBUTE void __tsan_write_range(void *addr, unsigned long size); // NOLINT @@ -123,6 +135,10 @@ int __tsan_get_report_loc(void *report, uptr idx, const char **type, int *fd, int *suppressable, void **trace, uptr trace_size); +SANITIZER_INTERFACE_ATTRIBUTE +int __tsan_get_report_loc_object_type(void *report, uptr idx, + const char **object_type); + // Returns information about mutexes included in the report. SANITIZER_INTERFACE_ATTRIBUTE int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr, @@ -130,7 +146,7 @@ int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr, // Returns information about threads included in the report. SANITIZER_INTERFACE_ATTRIBUTE -int __tsan_get_report_thread(void *report, uptr idx, int *tid, uptr *os_id, +int __tsan_get_report_thread(void *report, uptr idx, int *tid, tid_t *os_id, int *running, const char **name, int *parent_tid, void **trace, uptr trace_size); @@ -147,7 +163,7 @@ const char *__tsan_locate_address(uptr addr, char *name, uptr name_size, // Returns the allocation stack for a heap pointer. SANITIZER_INTERFACE_ATTRIBUTE int __tsan_get_alloc_stack(uptr addr, uptr *trace, uptr size, int *thread_id, - uptr *os_id); + tid_t *os_id); #endif // SANITIZER_GO diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc index 62db796..f68a0468 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc @@ -31,11 +31,10 @@ namespace __tsan { class ScopedAnnotation { public: - ScopedAnnotation(ThreadState *thr, const char *aname, const char *f, int l, - uptr pc) + ScopedAnnotation(ThreadState *thr, const char *aname, uptr pc) : thr_(thr) { FuncEntry(thr_, pc); - DPrintf("#%d: annotation %s() %s:%d\n", thr_->tid, aname, f, l); + DPrintf("#%d: annotation %s()\n", thr_->tid, aname); } ~ScopedAnnotation() { @@ -46,18 +45,20 @@ class ScopedAnnotation { ThreadState *const thr_; }; -#define SCOPED_ANNOTATION(typ) \ +#define SCOPED_ANNOTATION_RET(typ, ret) \ if (!flags()->enable_annotations) \ - return; \ + return ret; \ ThreadState *thr = cur_thread(); \ const uptr caller_pc = (uptr)__builtin_return_address(0); \ StatInc(thr, StatAnnotation); \ StatInc(thr, Stat##typ); \ - ScopedAnnotation sa(thr, __func__, f, l, caller_pc); \ + ScopedAnnotation sa(thr, __func__, caller_pc); \ const uptr pc = StackTrace::GetCurrentPc(); \ (void)pc; \ /**/ +#define SCOPED_ANNOTATION(typ) SCOPED_ANNOTATION_RET(typ, ) + static const int kMaxDescLen = 128; struct ExpectRace { @@ -252,12 +253,12 @@ void INTERFACE_ATTRIBUTE AnnotateCondVarWait(char *f, int l, uptr cv, void INTERFACE_ATTRIBUTE AnnotateRWLockCreate(char *f, int l, uptr m) { SCOPED_ANNOTATION(AnnotateRWLockCreate); - MutexCreate(thr, pc, m, true, true, false); + MutexCreate(thr, pc, m, MutexFlagWriteReentrant); } void INTERFACE_ATTRIBUTE AnnotateRWLockCreateStatic(char *f, int l, uptr m) { SCOPED_ANNOTATION(AnnotateRWLockCreateStatic); - MutexCreate(thr, pc, m, true, true, true); + MutexCreate(thr, pc, m, MutexFlagWriteReentrant | MutexFlagLinkerInit); } void INTERFACE_ATTRIBUTE AnnotateRWLockDestroy(char *f, int l, uptr m) { @@ -269,9 +270,9 @@ void INTERFACE_ATTRIBUTE AnnotateRWLockAcquired(char *f, int l, uptr m, uptr is_w) { SCOPED_ANNOTATION(AnnotateRWLockAcquired); if (is_w) - MutexLock(thr, pc, m); + MutexPostLock(thr, pc, m, MutexFlagDoPreLockOnPostLock); else - MutexReadLock(thr, pc, m); + MutexPostReadLock(thr, pc, m, MutexFlagDoPreLockOnPostLock); } void INTERFACE_ATTRIBUTE AnnotateRWLockReleased(char *f, int l, uptr m, @@ -458,4 +459,95 @@ void INTERFACE_ATTRIBUTE AnnotateMemoryIsInitialized(char *f, int l, uptr mem, uptr sz) {} void INTERFACE_ATTRIBUTE AnnotateMemoryIsUninitialized(char *f, int l, uptr mem, uptr sz) {} + +// Note: the parameter is called flagz, because flags is already taken +// by the global function that returns flags. +INTERFACE_ATTRIBUTE +void __tsan_mutex_create(void *m, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_create); + MutexCreate(thr, pc, (uptr)m, flagz & MutexCreationFlagMask); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_destroy(void *m, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_destroy); + MutexDestroy(thr, pc, (uptr)m, flagz); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_pre_lock(void *m, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_pre_lock); + if (!(flagz & MutexFlagTryLock)) { + if (flagz & MutexFlagReadLock) + MutexPreReadLock(thr, pc, (uptr)m); + else + MutexPreLock(thr, pc, (uptr)m); + } + ThreadIgnoreBegin(thr, pc, /*save_stack=*/false); + ThreadIgnoreSyncBegin(thr, pc, /*save_stack=*/false); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_post_lock(void *m, unsigned flagz, int rec) { + SCOPED_ANNOTATION(__tsan_mutex_post_lock); + ThreadIgnoreSyncEnd(thr, pc); + ThreadIgnoreEnd(thr, pc); + if (!(flagz & MutexFlagTryLockFailed)) { + if (flagz & MutexFlagReadLock) + MutexPostReadLock(thr, pc, (uptr)m, flagz); + else + MutexPostLock(thr, pc, (uptr)m, flagz, rec); + } +} + +INTERFACE_ATTRIBUTE +int __tsan_mutex_pre_unlock(void *m, unsigned flagz) { + SCOPED_ANNOTATION_RET(__tsan_mutex_pre_unlock, 0); + int ret = 0; + if (flagz & MutexFlagReadLock) { + CHECK(!(flagz & MutexFlagRecursiveUnlock)); + MutexReadUnlock(thr, pc, (uptr)m); + } else { + ret = MutexUnlock(thr, pc, (uptr)m, flagz); + } + ThreadIgnoreBegin(thr, pc, /*save_stack=*/false); + ThreadIgnoreSyncBegin(thr, pc, /*save_stack=*/false); + return ret; +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_post_unlock(void *m, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_post_unlock); + ThreadIgnoreSyncEnd(thr, pc); + ThreadIgnoreEnd(thr, pc); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_pre_signal(void *addr, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_pre_signal); + ThreadIgnoreBegin(thr, pc, /*save_stack=*/false); + ThreadIgnoreSyncBegin(thr, pc, /*save_stack=*/false); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_post_signal(void *addr, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_post_signal); + ThreadIgnoreSyncEnd(thr, pc); + ThreadIgnoreEnd(thr, pc); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_pre_divert(void *addr, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_pre_divert); + // Exit from ignore region started in __tsan_mutex_pre_lock/unlock/signal. + ThreadIgnoreSyncEnd(thr, pc); + ThreadIgnoreEnd(thr, pc); +} + +INTERFACE_ATTRIBUTE +void __tsan_mutex_post_divert(void *addr, unsigned flagz) { + SCOPED_ANNOTATION(__tsan_mutex_post_divert); + ThreadIgnoreBegin(thr, pc, /*save_stack=*/false); + ThreadIgnoreSyncBegin(thr, pc, /*save_stack=*/false); +} } // extern "C" diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc index 5238b66..d334394 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc @@ -220,8 +220,7 @@ static a128 NoTsanAtomicLoad(const volatile a128 *a, morder mo) { #endif template<typename T> -static T AtomicLoad(ThreadState *thr, uptr pc, const volatile T *a, - morder mo) { +static T AtomicLoad(ThreadState *thr, uptr pc, const volatile T *a, morder mo) { CHECK(IsLoadOrder(mo)); // This fast-path is critical for performance. // Assume the access is atomic. @@ -229,10 +228,17 @@ static T AtomicLoad(ThreadState *thr, uptr pc, const volatile T *a, MemoryReadAtomic(thr, pc, (uptr)a, SizeLog<T>()); return NoTsanAtomicLoad(a, mo); } - SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, (uptr)a, false); - AcquireImpl(thr, pc, &s->clock); + // Don't create sync object if it does not exist yet. For example, an atomic + // pointer is initialized to nullptr and then periodically acquire-loaded. T v = NoTsanAtomicLoad(a, mo); - s->mtx.ReadUnlock(); + SyncVar *s = ctx->metamap.GetIfExistsAndLock((uptr)a, false); + if (s) { + AcquireImpl(thr, pc, &s->clock); + // Re-read under sync mutex because we need a consistent snapshot + // of the value and the clock we acquire. + v = NoTsanAtomicLoad(a, mo); + s->mtx.ReadUnlock(); + } MemoryReadAtomic(thr, pc, (uptr)a, SizeLog<T>()); return v; } @@ -450,13 +456,32 @@ static void AtomicFence(ThreadState *thr, uptr pc, morder mo) { // C/C++ +static morder convert_morder(morder mo) { + if (flags()->force_seq_cst_atomics) + return (morder)mo_seq_cst; + + // Filter out additional memory order flags: + // MEMMODEL_SYNC = 1 << 15 + // __ATOMIC_HLE_ACQUIRE = 1 << 16 + // __ATOMIC_HLE_RELEASE = 1 << 17 + // + // HLE is an optimization, and we pretend that elision always fails. + // MEMMODEL_SYNC is used when lowering __sync_ atomics, + // since we use __sync_ atomics for actual atomic operations, + // we can safely ignore it as well. It also subtly affects semantics, + // but we don't model the difference. + return (morder)(mo & 0x7fff); +} + #define SCOPED_ATOMIC(func, ...) \ - const uptr callpc = (uptr)__builtin_return_address(0); \ - uptr pc = StackTrace::GetCurrentPc(); \ - mo = flags()->force_seq_cst_atomics ? (morder)mo_seq_cst : mo; \ ThreadState *const thr = cur_thread(); \ - if (thr->ignore_interceptors) \ + if (thr->ignore_sync || thr->ignore_interceptors) { \ + ProcessPendingSignals(thr); \ return NoTsanAtomic##func(__VA_ARGS__); \ + } \ + const uptr callpc = (uptr)__builtin_return_address(0); \ + uptr pc = StackTrace::GetCurrentPc(); \ + mo = convert_morder(mo); \ AtomicStatInc(thr, sizeof(*a), mo, StatAtomic##func); \ ScopedAtomic sa(thr, callpc, a, mo, __func__); \ return Atomic##func(thr, pc, __VA_ARGS__); \ diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_java.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_java.cc index 5bdc04f..75e960e 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_java.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_interface_java.cc @@ -180,8 +180,8 @@ void __tsan_java_mutex_lock(jptr addr) { CHECK_GE(addr, jctx->heap_begin); CHECK_LT(addr, jctx->heap_begin + jctx->heap_size); - MutexCreate(thr, pc, addr, true, true, true); - MutexLock(thr, pc, addr); + MutexPostLock(thr, pc, addr, MutexFlagLinkerInit | MutexFlagWriteReentrant | + MutexFlagDoPreLockOnPostLock); } void __tsan_java_mutex_unlock(jptr addr) { @@ -201,8 +201,8 @@ void __tsan_java_mutex_read_lock(jptr addr) { CHECK_GE(addr, jctx->heap_begin); CHECK_LT(addr, jctx->heap_begin + jctx->heap_size); - MutexCreate(thr, pc, addr, true, true, true); - MutexReadLock(thr, pc, addr); + MutexPostReadLock(thr, pc, addr, MutexFlagLinkerInit | + MutexFlagWriteReentrant | MutexFlagDoPreLockOnPostLock); } void __tsan_java_mutex_read_unlock(jptr addr) { @@ -223,8 +223,8 @@ void __tsan_java_mutex_lock_rec(jptr addr, int rec) { CHECK_LT(addr, jctx->heap_begin + jctx->heap_size); CHECK_GT(rec, 0); - MutexCreate(thr, pc, addr, true, true, true); - MutexLock(thr, pc, addr, rec); + MutexPostLock(thr, pc, addr, MutexFlagLinkerInit | MutexFlagWriteReentrant | + MutexFlagDoPreLockOnPostLock | MutexFlagRecursiveLock, rec); } int __tsan_java_mutex_unlock_rec(jptr addr) { @@ -234,7 +234,7 @@ int __tsan_java_mutex_unlock_rec(jptr addr) { CHECK_GE(addr, jctx->heap_begin); CHECK_LT(addr, jctx->heap_begin + jctx->heap_size); - return MutexUnlock(thr, pc, addr, true); + return MutexUnlock(thr, pc, addr, MutexFlagRecursiveUnlock); } void __tsan_java_acquire(jptr addr) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_libdispatch_mac.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_libdispatch_mac.cc index d8c689e..8c759a3 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_libdispatch_mac.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_libdispatch_mac.cc @@ -93,14 +93,15 @@ static tsan_block_context_t *AllocContext(ThreadState *thr, uptr pc, new_context->free_context_in_callback = true; new_context->submitted_synchronously = false; new_context->is_barrier_block = false; + new_context->non_queue_sync_object = 0; return new_context; } -#define GET_QUEUE_SYNC_VARS(context, q) \ - bool is_queue_serial = q && IsQueueSerial(q); \ - uptr sync_ptr = (uptr)q ?: context->non_queue_sync_object; \ - uptr serial_sync = (uptr)sync_ptr; \ - uptr concurrent_sync = ((uptr)sync_ptr) + sizeof(uptr); \ +#define GET_QUEUE_SYNC_VARS(context, q) \ + bool is_queue_serial = q && IsQueueSerial(q); \ + uptr sync_ptr = (uptr)q ?: context->non_queue_sync_object; \ + uptr serial_sync = (uptr)sync_ptr; \ + uptr concurrent_sync = sync_ptr ? ((uptr)sync_ptr) + sizeof(uptr) : 0; \ bool serial_task = context->is_barrier_block || is_queue_serial static void dispatch_sync_pre_execute(ThreadState *thr, uptr pc, @@ -111,8 +112,8 @@ static void dispatch_sync_pre_execute(ThreadState *thr, uptr pc, dispatch_queue_t q = context->queue; do { GET_QUEUE_SYNC_VARS(context, q); - Acquire(thr, pc, serial_sync); - if (serial_task) Acquire(thr, pc, concurrent_sync); + if (serial_sync) Acquire(thr, pc, serial_sync); + if (serial_task && concurrent_sync) Acquire(thr, pc, concurrent_sync); if (q) q = GetTargetQueueFromQueue(q); } while (q); @@ -126,7 +127,8 @@ static void dispatch_sync_post_execute(ThreadState *thr, uptr pc, dispatch_queue_t q = context->queue; do { GET_QUEUE_SYNC_VARS(context, q); - Release(thr, pc, serial_task ? serial_sync : concurrent_sync); + if (serial_task && serial_sync) Release(thr, pc, serial_sync); + if (!serial_task && concurrent_sync) Release(thr, pc, concurrent_sync); if (q) q = GetTargetQueueFromQueue(q); } while (q); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_mman.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_mman.cc index 2dea249..f79dccd 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_mman.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_mman.cc @@ -10,6 +10,7 @@ // This file is a part of ThreadSanitizer (TSan), a race detector. // //===----------------------------------------------------------------------===// +#include "sanitizer_common/sanitizer_allocator_checks.h" #include "sanitizer_common/sanitizer_allocator_interface.h" #include "sanitizer_common/sanitizer_common.h" #include "sanitizer_common/sanitizer_placement_new.h" @@ -112,9 +113,8 @@ ScopedGlobalProcessor::~ScopedGlobalProcessor() { } void InitializeAllocator() { - allocator()->Init( - common_flags()->allocator_may_return_null, - common_flags()->allocator_release_to_os_interval_ms); + SetAllocatorMayReturnNull(common_flags()->allocator_may_return_null); + allocator()->Init(common_flags()->allocator_release_to_os_interval_ms); } void InitializeAllocatorLate() { @@ -151,7 +151,7 @@ static void SignalUnsafeCall(ThreadState *thr, uptr pc) { void *user_alloc(ThreadState *thr, uptr pc, uptr sz, uptr align, bool signal) { if ((sz >= (1ull << 40)) || (align >= (1ull << 40))) - return allocator()->ReturnNullOrDieOnBadRequest(); + return Allocator::FailureHandler::OnBadRequest(); void *p = allocator()->Allocate(&thr->proc()->alloc_cache, sz, align); if (p == 0) return 0; @@ -163,8 +163,8 @@ void *user_alloc(ThreadState *thr, uptr pc, uptr sz, uptr align, bool signal) { } void *user_calloc(ThreadState *thr, uptr pc, uptr size, uptr n) { - if (CallocShouldReturnNullDueToOverflow(size, n)) - return allocator()->ReturnNullOrDieOnBadRequest(); + if (CheckForCallocOverflow(size, n)) + return Allocator::FailureHandler::OnBadRequest(); void *p = user_alloc(thr, pc, n * size); if (p) internal_memset(p, 0, n * size); @@ -295,6 +295,8 @@ uptr __sanitizer_get_allocated_size(const void *p) { void __tsan_on_thread_idle() { ThreadState *thr = cur_thread(); + thr->clock.ResetCached(&thr->proc()->clock_cache); + thr->last_sleep_clock.ResetCached(&thr->proc()->clock_cache); allocator()->SwallowCache(&thr->proc()->alloc_cache); internal_allocator()->SwallowCache(&thr->proc()->internal_alloc_cache); ctx->metamap.OnProcIdle(thr->proc()); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc index b6478bb..4d03145 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc @@ -12,6 +12,7 @@ // Interceptors for operators new and delete. //===----------------------------------------------------------------------===// #include "interception/interception.h" +#include "sanitizer_common/sanitizer_allocator.h" #include "sanitizer_common/sanitizer_internal_defs.h" #include "tsan_interceptors.h" @@ -24,13 +25,15 @@ struct nothrow_t {}; DECLARE_REAL(void *, malloc, uptr size) DECLARE_REAL(void, free, void *ptr) -#define OPERATOR_NEW_BODY(mangled_name) \ +// TODO(alekseys): throw std::bad_alloc instead of dying on OOM. +#define OPERATOR_NEW_BODY(mangled_name, nothrow) \ if (cur_thread()->in_symbolizer) \ return InternalAlloc(size); \ void *p = 0; \ { \ SCOPED_INTERCEPTOR_RAW(mangled_name, size); \ p = user_alloc(thr, pc, size); \ + if (!nothrow && UNLIKELY(!p)) DieOnFailure::OnOOM(); \ } \ invoke_malloc_hook(p, size); \ return p; @@ -38,25 +41,25 @@ DECLARE_REAL(void, free, void *ptr) SANITIZER_INTERFACE_ATTRIBUTE void *operator new(__sanitizer::uptr size); void *operator new(__sanitizer::uptr size) { - OPERATOR_NEW_BODY(_Znwm); + OPERATOR_NEW_BODY(_Znwm, false /*nothrow*/); } SANITIZER_INTERFACE_ATTRIBUTE void *operator new[](__sanitizer::uptr size); void *operator new[](__sanitizer::uptr size) { - OPERATOR_NEW_BODY(_Znam); + OPERATOR_NEW_BODY(_Znam, false /*nothrow*/); } SANITIZER_INTERFACE_ATTRIBUTE void *operator new(__sanitizer::uptr size, std::nothrow_t const&); void *operator new(__sanitizer::uptr size, std::nothrow_t const&) { - OPERATOR_NEW_BODY(_ZnwmRKSt9nothrow_t); + OPERATOR_NEW_BODY(_ZnwmRKSt9nothrow_t, true /*nothrow*/); } SANITIZER_INTERFACE_ATTRIBUTE void *operator new[](__sanitizer::uptr size, std::nothrow_t const&); void *operator new[](__sanitizer::uptr size, std::nothrow_t const&) { - OPERATOR_NEW_BODY(_ZnamRKSt9nothrow_t); + OPERATOR_NEW_BODY(_ZnamRKSt9nothrow_t, true /*nothrow*/); } #define OPERATOR_DELETE_BODY(mangled_name) \ diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform.h index 1dd9d91..bea1dab 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform.h @@ -100,6 +100,37 @@ struct Mapping { }; #define TSAN_MID_APP_RANGE 1 +#elif defined(__aarch64__) && defined(__APPLE__) +/* +C/C++ on Darwin/iOS/ARM64 (36-bit VMA, 64 GB VM) +0000 0000 00 - 0100 0000 00: - (4 GB) +0100 0000 00 - 0200 0000 00: main binary, modules, thread stacks (4 GB) +0200 0000 00 - 0300 0000 00: heap (4 GB) +0300 0000 00 - 0400 0000 00: - (4 GB) +0400 0000 00 - 0c00 0000 00: shadow memory (32 GB) +0c00 0000 00 - 0d00 0000 00: - (4 GB) +0d00 0000 00 - 0e00 0000 00: metainfo (4 GB) +0e00 0000 00 - 0f00 0000 00: - (4 GB) +0f00 0000 00 - 1000 0000 00: traces (4 GB) +*/ +struct Mapping { + static const uptr kLoAppMemBeg = 0x0100000000ull; + static const uptr kLoAppMemEnd = 0x0200000000ull; + static const uptr kHeapMemBeg = 0x0200000000ull; + static const uptr kHeapMemEnd = 0x0300000000ull; + static const uptr kShadowBeg = 0x0400000000ull; + static const uptr kShadowEnd = 0x0c00000000ull; + static const uptr kMetaShadowBeg = 0x0d00000000ull; + static const uptr kMetaShadowEnd = 0x0e00000000ull; + static const uptr kTraceMemBeg = 0x0f00000000ull; + static const uptr kTraceMemEnd = 0x1000000000ull; + static const uptr kHiAppMemBeg = 0x1000000000ull; + static const uptr kHiAppMemEnd = 0x1000000000ull; + static const uptr kAppMemMsk = 0x0ull; + static const uptr kAppMemXor = 0x0ull; + static const uptr kVdsoBeg = 0x7000000000000000ull; +}; + #elif defined(__aarch64__) // AArch64 supports multiple VMA which leads to multiple address transformation // functions. To support these multiple VMAS transformations and mappings TSAN @@ -389,7 +420,7 @@ uptr MappingImpl(void) { template<int Type> uptr MappingArchImpl(void) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return MappingImpl<Mapping39, Type>(); case 42: return MappingImpl<Mapping42, Type>(); @@ -542,7 +573,7 @@ bool IsAppMemImpl(uptr mem) { ALWAYS_INLINE bool IsAppMem(uptr mem) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return IsAppMemImpl<Mapping39>(mem); case 42: return IsAppMemImpl<Mapping42>(mem); @@ -569,7 +600,7 @@ bool IsShadowMemImpl(uptr mem) { ALWAYS_INLINE bool IsShadowMem(uptr mem) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return IsShadowMemImpl<Mapping39>(mem); case 42: return IsShadowMemImpl<Mapping42>(mem); @@ -596,7 +627,7 @@ bool IsMetaMemImpl(uptr mem) { ALWAYS_INLINE bool IsMetaMem(uptr mem) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return IsMetaMemImpl<Mapping39>(mem); case 42: return IsMetaMemImpl<Mapping42>(mem); @@ -633,7 +664,7 @@ uptr MemToShadowImpl(uptr x) { ALWAYS_INLINE uptr MemToShadow(uptr x) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return MemToShadowImpl<Mapping39>(x); case 42: return MemToShadowImpl<Mapping42>(x); @@ -672,7 +703,7 @@ u32 *MemToMetaImpl(uptr x) { ALWAYS_INLINE u32 *MemToMeta(uptr x) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return MemToMetaImpl<Mapping39>(x); case 42: return MemToMetaImpl<Mapping42>(x); @@ -724,7 +755,7 @@ uptr ShadowToMemImpl(uptr s) { ALWAYS_INLINE uptr ShadowToMem(uptr s) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return ShadowToMemImpl<Mapping39>(s); case 42: return ShadowToMemImpl<Mapping42>(s); @@ -759,7 +790,7 @@ uptr GetThreadTraceImpl(int tid) { ALWAYS_INLINE uptr GetThreadTrace(int tid) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return GetThreadTraceImpl<Mapping39>(tid); case 42: return GetThreadTraceImpl<Mapping42>(tid); @@ -789,7 +820,7 @@ uptr GetThreadTraceHeaderImpl(int tid) { ALWAYS_INLINE uptr GetThreadTraceHeader(int tid) { -#ifdef __aarch64__ +#if defined(__aarch64__) && !defined(__APPLE__) switch (vmaSize) { case 39: return GetThreadTraceHeaderImpl<Mapping39>(tid); case 42: return GetThreadTraceHeaderImpl<Mapping42>(tid); @@ -816,6 +847,7 @@ void FlushShadowMemory(); void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive); int ExtractResolvFDs(void *state, int *fds, int nfd); int ExtractRecvmsgFDs(void *msg, int *fds, int nfd); +void ImitateTlsWrite(ThreadState *thr, uptr tls_addr, uptr tls_size); int call_pthread_cancel_with_cleanup(int(*fn)(void *c, void *m, void *abstime), void *c, void *m, void *abstime, diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc index 3313288..ead1e57 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc @@ -47,7 +47,6 @@ #include <sys/resource.h> #include <sys/stat.h> #include <unistd.h> -#include <errno.h> #include <sched.h> #include <dlfcn.h> #if SANITIZER_LINUX @@ -182,17 +181,15 @@ static void MapRodata() { } // Map the file into shadow of .rodata sections. MemoryMappingLayout proc_maps(/*cache_enabled*/true); - uptr start, end, offset, prot; // Reusing the buffer 'name'. - while (proc_maps.Next(&start, &end, &offset, name, ARRAY_SIZE(name), &prot)) { - if (name[0] != 0 && name[0] != '[' - && (prot & MemoryMappingLayout::kProtectionRead) - && (prot & MemoryMappingLayout::kProtectionExecute) - && !(prot & MemoryMappingLayout::kProtectionWrite) - && IsAppMem(start)) { + MemoryMappedSegment segment(name, ARRAY_SIZE(name)); + while (proc_maps.Next(&segment)) { + if (segment.filename[0] != 0 && segment.filename[0] != '[' && + segment.IsReadable() && segment.IsExecutable() && + !segment.IsWritable() && IsAppMem(segment.start)) { // Assume it's .rodata - char *shadow_start = (char*)MemToShadow(start); - char *shadow_end = (char*)MemToShadow(end); + char *shadow_start = (char *)MemToShadow(segment.start); + char *shadow_end = (char *)MemToShadow(segment.end); for (char *p = shadow_start; p < shadow_end; p += marker.size()) { internal_mmap(p, Min<uptr>(marker.size(), shadow_end - p), PROT_READ, MAP_PRIVATE | MAP_FIXED, fd, 0); @@ -289,7 +286,7 @@ void InitializePlatform() { int ExtractResolvFDs(void *state, int *fds, int nfd) { #if SANITIZER_LINUX && !SANITIZER_ANDROID int cnt = 0; - __res_state *statp = (__res_state*)state; + struct __res_state *statp = (struct __res_state*)state; for (int i = 0; i < MAXNS && cnt < nfd; i++) { if (statp->_u._ext.nsaddrs[i] && statp->_u._ext.nssocks[i] != -1) fds[cnt++] = statp->_u._ext.nssocks[i]; @@ -320,6 +317,20 @@ int ExtractRecvmsgFDs(void *msgp, int *fds, int nfd) { return res; } +void ImitateTlsWrite(ThreadState *thr, uptr tls_addr, uptr tls_size) { + // Check that the thr object is in tls; + const uptr thr_beg = (uptr)thr; + const uptr thr_end = (uptr)thr + sizeof(*thr); + CHECK_GE(thr_beg, tls_addr); + CHECK_LE(thr_beg, tls_addr + tls_size); + CHECK_GE(thr_end, tls_addr); + CHECK_LE(thr_end, tls_addr + tls_size); + // Since the thr object is huge, skip it. + MemoryRangeImitateWrite(thr, /*pc=*/2, tls_addr, thr_beg - tls_addr); + MemoryRangeImitateWrite(thr, /*pc=*/2, thr_end, + tls_addr + tls_size - thr_end); +} + // Note: this function runs with async signals enabled, // so it must not touch any tsan state. int call_pthread_cancel_with_cleanup(int(*fn)(void *c, void *m, @@ -341,36 +352,22 @@ void ReplaceSystemMalloc() { } #if !SANITIZER_GO #if SANITIZER_ANDROID - -#if defined(__aarch64__) -# define __get_tls() \ - ({ void** __val; __asm__("mrs %0, tpidr_el0" : "=r"(__val)); __val; }) -#elif defined(__x86_64__) -# define __get_tls() \ - ({ void** __val; __asm__("mov %%fs:0, %0" : "=r"(__val)); __val; }) -#else -#error unsupported architecture -#endif - -// On Android, __thread is not supported. So we store the pointer to ThreadState -// in TLS_SLOT_TSAN, which is the tls slot allocated by Android bionic for tsan. -static const int TLS_SLOT_TSAN = 8; // On Android, one thread can call intercepted functions after // DestroyThreadState(), so add a fake thread state for "dead" threads. static ThreadState *dead_thread_state = nullptr; ThreadState *cur_thread() { - ThreadState* thr = (ThreadState*)__get_tls()[TLS_SLOT_TSAN]; + ThreadState* thr = reinterpret_cast<ThreadState*>(*get_android_tls_ptr()); if (thr == nullptr) { __sanitizer_sigset_t emptyset; internal_sigfillset(&emptyset); __sanitizer_sigset_t oldset; CHECK_EQ(0, internal_sigprocmask(SIG_SETMASK, &emptyset, &oldset)); - thr = reinterpret_cast<ThreadState*>(__get_tls()[TLS_SLOT_TSAN]); + thr = reinterpret_cast<ThreadState*>(*get_android_tls_ptr()); if (thr == nullptr) { thr = reinterpret_cast<ThreadState*>(MmapOrDie(sizeof(ThreadState), "ThreadState")); - __get_tls()[TLS_SLOT_TSAN] = thr; + *get_android_tls_ptr() = reinterpret_cast<uptr>(thr); if (dead_thread_state == nullptr) { dead_thread_state = reinterpret_cast<ThreadState*>( MmapOrDie(sizeof(ThreadState), "ThreadState")); @@ -392,9 +389,9 @@ void cur_thread_finalize() { internal_sigfillset(&emptyset); __sanitizer_sigset_t oldset; CHECK_EQ(0, internal_sigprocmask(SIG_SETMASK, &emptyset, &oldset)); - ThreadState* thr = (ThreadState*)__get_tls()[TLS_SLOT_TSAN]; + ThreadState* thr = reinterpret_cast<ThreadState*>(*get_android_tls_ptr()); if (thr != dead_thread_state) { - __get_tls()[TLS_SLOT_TSAN] = dead_thread_state; + *get_android_tls_ptr() = reinterpret_cast<uptr>(dead_thread_state); UnmapOrDie(thr, sizeof(ThreadState)); } CHECK_EQ(0, internal_sigprocmask(SIG_SETMASK, &oldset, nullptr)); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_mac.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_mac.cc index 25dd241..73a656f 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_mac.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_mac.cc @@ -75,12 +75,18 @@ static void *SignalSafeGetOrAllocate(uptr *dst, uptr size) { static uptr main_thread_identity = 0; ALIGNED(64) static char main_thread_state[sizeof(ThreadState)]; +ThreadState **cur_thread_location() { + ThreadState **thread_identity = (ThreadState **)pthread_self(); + return ((uptr)thread_identity == main_thread_identity) ? nullptr + : thread_identity; +} + ThreadState *cur_thread() { - uptr thread_identity = (uptr)pthread_self(); - if (thread_identity == main_thread_identity || main_thread_identity == 0) { + ThreadState **thr_state_loc = cur_thread_location(); + if (thr_state_loc == nullptr || main_thread_identity == 0) { return (ThreadState *)&main_thread_state; } - ThreadState **fake_tls = (ThreadState **)MemToShadow(thread_identity); + ThreadState **fake_tls = (ThreadState **)MemToShadow((uptr)thr_state_loc); ThreadState *thr = (ThreadState *)SignalSafeGetOrAllocate( (uptr *)fake_tls, sizeof(ThreadState)); return thr; @@ -90,13 +96,13 @@ ThreadState *cur_thread() { // munmap first and then clear `fake_tls`; if we receive a signal in between, // handler will try to access the unmapped ThreadState. void cur_thread_finalize() { - uptr thread_identity = (uptr)pthread_self(); - if (thread_identity == main_thread_identity) { + ThreadState **thr_state_loc = cur_thread_location(); + if (thr_state_loc == nullptr) { // Calling dispatch_main() or xpc_main() actually invokes pthread_exit to // exit the main thread. Let's keep the main thread's ThreadState. return; } - ThreadState **fake_tls = (ThreadState **)MemToShadow(thread_identity); + ThreadState **fake_tls = (ThreadState **)MemToShadow((uptr)thr_state_loc); internal_munmap(*fake_tls, sizeof(ThreadState)); *fake_tls = nullptr; } @@ -207,7 +213,7 @@ static void my_pthread_introspection_hook(unsigned int event, pthread_t thread, ThreadState *parent_thread_state = nullptr; // No parent. int tid = ThreadCreate(parent_thread_state, 0, (uptr)thread, true); CHECK_NE(tid, 0); - ThreadStart(thr, tid, GetTid()); + ThreadStart(thr, tid, GetTid(), /*workerthread*/ true); } } else if (event == PTHREAD_INTROSPECTION_THREAD_TERMINATE) { if (thread == pthread_self()) { @@ -224,6 +230,14 @@ static void my_pthread_introspection_hook(unsigned int event, pthread_t thread, #endif void InitializePlatformEarly() { +#if defined(__aarch64__) + uptr max_vm = GetMaxVirtualAddress() + 1; + if (max_vm != Mapping::kHiAppMemEnd) { + Printf("ThreadSanitizer: unsupported vm address limit %p, expected %p.\n", + max_vm, Mapping::kHiAppMemEnd); + Die(); + } +#endif } void InitializePlatform() { @@ -240,6 +254,29 @@ void InitializePlatform() { } #if !SANITIZER_GO +void ImitateTlsWrite(ThreadState *thr, uptr tls_addr, uptr tls_size) { + // The pointer to the ThreadState object is stored in the shadow memory + // of the tls. + uptr tls_end = tls_addr + tls_size; + ThreadState **thr_state_loc = cur_thread_location(); + if (thr_state_loc == nullptr) { + MemoryRangeImitateWrite(thr, /*pc=*/2, tls_addr, tls_size); + } else { + uptr thr_state_start = (uptr)thr_state_loc; + uptr thr_state_end = thr_state_start + sizeof(uptr); + CHECK_GE(thr_state_start, tls_addr); + CHECK_LE(thr_state_start, tls_addr + tls_size); + CHECK_GE(thr_state_end, tls_addr); + CHECK_LE(thr_state_end, tls_addr + tls_size); + MemoryRangeImitateWrite(thr, /*pc=*/2, tls_addr, + thr_state_start - tls_addr); + MemoryRangeImitateWrite(thr, /*pc=*/2, thr_state_end, + tls_end - thr_state_end); + } +} +#endif + +#if !SANITIZER_GO // Note: this function runs with async signals enabled, // so it must not touch any tsan state. int call_pthread_cancel_with_cleanup(int(*fn)(void *c, void *m, diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_posix.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_posix.cc index 0732c83..e4f90a8 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_posix.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_posix.cc @@ -46,6 +46,9 @@ void InitializeShadowMemory() { #elif defined(__mips64) const uptr kMadviseRangeBeg = 0xff00000000ull; const uptr kMadviseRangeSize = 0x0100000000ull; +#elif defined(__aarch64__) && defined(__APPLE__) + uptr kMadviseRangeBeg = LoAppMemBeg(); + uptr kMadviseRangeSize = LoAppMemEnd() - LoAppMemBeg(); #elif defined(__aarch64__) uptr kMadviseRangeBeg = 0; uptr kMadviseRangeSize = 0; @@ -115,21 +118,24 @@ static void ProtectRange(uptr beg, uptr end) { void CheckAndProtect() { // Ensure that the binary is indeed compiled with -pie. MemoryMappingLayout proc_maps(true); - uptr p, end, prot; - while (proc_maps.Next(&p, &end, 0, 0, 0, &prot)) { - if (IsAppMem(p)) + MemoryMappedSegment segment; + while (proc_maps.Next(&segment)) { + if (IsAppMem(segment.start)) continue; + if (segment.start >= HeapMemEnd() && segment.start < HeapEnd()) continue; + if (segment.protection == 0) // Zero page or mprotected. continue; - if (p >= HeapMemEnd() && - p < HeapEnd()) - continue; - if (prot == 0) // Zero page or mprotected. - continue; - if (p >= VdsoBeg()) // vdso + if (segment.start >= VdsoBeg()) // vdso break; - Printf("FATAL: ThreadSanitizer: unexpected memory mapping %p-%p\n", p, end); + Printf("FATAL: ThreadSanitizer: unexpected memory mapping %p-%p\n", + segment.start, segment.end); Die(); } +#if defined(__aarch64__) && defined(__APPLE__) + ProtectRange(HeapMemEnd(), ShadowBeg()); + ProtectRange(ShadowEnd(), MetaShadowBeg()); + ProtectRange(MetaShadowEnd(), TraceMemBeg()); +#else ProtectRange(LoAppMemEnd(), ShadowBeg()); ProtectRange(ShadowEnd(), MetaShadowBeg()); #ifdef TSAN_MID_APP_RANGE @@ -143,6 +149,7 @@ void CheckAndProtect() { ProtectRange(TraceMemBeg(), TraceMemEnd()); ProtectRange(TraceMemEnd(), HeapMemBeg()); ProtectRange(HeapEnd(), HiAppMemBeg()); +#endif } #endif diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_report.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_report.cc index 07fd412..32cc332 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_report.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_report.cc @@ -53,7 +53,8 @@ class Decorator: public __sanitizer::SanitizerCommonDecorator { }; ReportDesc::ReportDesc() - : stacks(MBlockReportStack) + : tag(kExternalTagNone) + , stacks(MBlockReportStack) , mops(MBlockReportMop) , locs(MBlockReportLoc) , mutexes(MBlockReportMutex) @@ -81,7 +82,7 @@ const char *thread_name(char *buf, int tid) { return buf; } -static const char *ReportTypeString(ReportType typ) { +static const char *ReportTypeString(ReportType typ, uptr tag) { if (typ == ReportTypeRace) return "data race"; if (typ == ReportTypeVptrRace) @@ -90,6 +91,10 @@ static const char *ReportTypeString(ReportType typ) { return "heap-use-after-free"; if (typ == ReportTypeVptrUseAfterFree) return "heap-use-after-free (virtual call vs free)"; + if (typ == ReportTypeExternalRace) { + const char *str = GetReportHeaderFromTag(tag); + return str ? str : "race on external object"; + } if (typ == ReportTypeThreadLeak) return "thread leak"; if (typ == ReportTypeMutexDestroyLocked) @@ -152,14 +157,27 @@ static const char *MopDesc(bool first, bool write, bool atomic) { : (write ? "Previous write" : "Previous read")); } +static const char *ExternalMopDesc(bool first, bool write) { + return first ? (write ? "Modifying" : "Read-only") + : (write ? "Previous modifying" : "Previous read-only"); +} + static void PrintMop(const ReportMop *mop, bool first) { Decorator d; char thrbuf[kThreadBufSize]; Printf("%s", d.Access()); - Printf(" %s of size %d at %p by %s", - MopDesc(first, mop->write, mop->atomic), - mop->size, (void*)mop->addr, - thread_name(thrbuf, mop->tid)); + if (mop->external_tag == kExternalTagNone) { + Printf(" %s of size %d at %p by %s", + MopDesc(first, mop->write, mop->atomic), mop->size, + (void *)mop->addr, thread_name(thrbuf, mop->tid)); + } else { + const char *object_type = GetObjectTypeFromTag(mop->external_tag); + if (object_type == nullptr) + object_type = "external object"; + Printf(" %s access of %s at %p by %s", + ExternalMopDesc(first, mop->write), object_type, + (void *)mop->addr, thread_name(thrbuf, mop->tid)); + } PrintMutexSet(mop->mset); Printf(":\n"); Printf("%s", d.EndAccess()); @@ -183,9 +201,16 @@ static void PrintLocation(const ReportLocation *loc) { global.module_offset); } else if (loc->type == ReportLocationHeap) { char thrbuf[kThreadBufSize]; - Printf(" Location is heap block of size %zu at %p allocated by %s:\n", - loc->heap_chunk_size, loc->heap_chunk_start, - thread_name(thrbuf, loc->tid)); + const char *object_type = GetObjectTypeFromTag(loc->external_tag); + if (!object_type) { + Printf(" Location is heap block of size %zu at %p allocated by %s:\n", + loc->heap_chunk_size, loc->heap_chunk_start, + thread_name(thrbuf, loc->tid)); + } else { + Printf(" Location is %s of size %zu at %p allocated by %s:\n", + object_type, loc->heap_chunk_size, loc->heap_chunk_start, + thread_name(thrbuf, loc->tid)); + } print_stack = true; } else if (loc->type == ReportLocationStack) { Printf(" Location is stack of %s.\n\n", thread_name(thrbuf, loc->tid)); @@ -235,9 +260,15 @@ static void PrintThread(const ReportThread *rt) { if (rt->name && rt->name[0] != '\0') Printf(" '%s'", rt->name); char thrbuf[kThreadBufSize]; - Printf(" (tid=%zu, %s) created by %s", - rt->os_id, rt->running ? "running" : "finished", - thread_name(thrbuf, rt->parent_tid)); + const char *thread_status = rt->running ? "running" : "finished"; + if (rt->workerthread) { + Printf(" (tid=%zu, %s) is a GCD worker thread\n", rt->os_id, thread_status); + Printf("\n"); + Printf("%s", d.EndThreadDescription()); + return; + } + Printf(" (tid=%zu, %s) created by %s", rt->os_id, thread_status, + thread_name(thrbuf, rt->parent_tid)); if (rt->stack) Printf(" at:"); Printf("\n"); @@ -289,7 +320,7 @@ static SymbolizedStack *SkipTsanInternalFrames(SymbolizedStack *frames) { void PrintReport(const ReportDesc *rep) { Decorator d; Printf("==================\n"); - const char *rep_typ_str = ReportTypeString(rep->typ); + const char *rep_typ_str = ReportTypeString(rep->typ, rep->tag); Printf("%s", d.Warning()); Printf("WARNING: ThreadSanitizer: %s (pid=%d)\n", rep_typ_str, (int)internal_getpid()); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_report.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_report.h index d0b9d74..bc1582f 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_report.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_report.h @@ -24,6 +24,7 @@ enum ReportType { ReportTypeVptrRace, ReportTypeUseAfterFree, ReportTypeVptrUseAfterFree, + ReportTypeExternalRace, ReportTypeThreadLeak, ReportTypeMutexDestroyLocked, ReportTypeMutexDoubleLock, @@ -56,6 +57,7 @@ struct ReportMop { int size; bool write; bool atomic; + uptr external_tag; Vector<ReportMopMutex> mset; ReportStack *stack; @@ -75,6 +77,7 @@ struct ReportLocation { DataInfo global; uptr heap_chunk_start; uptr heap_chunk_size; + uptr external_tag; int tid; int fd; bool suppressable; @@ -87,10 +90,11 @@ struct ReportLocation { struct ReportThread { int id; - uptr os_id; + tid_t os_id; bool running; + bool workerthread; char *name; - int parent_tid; + u32 parent_tid; ReportStack *stack; }; @@ -104,6 +108,7 @@ struct ReportMutex { class ReportDesc { public: ReportType typ; + uptr tag; Vector<ReportStack*> stacks; Vector<ReportMop*> mops; Vector<ReportLocation*> locs; diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.cc index bfb8358..a015253 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.cc @@ -104,7 +104,8 @@ Context::Context() , racy_stacks(MBlockRacyStacks) , racy_addresses(MBlockRacyAddresses) , fired_suppressions_mtx(MutexTypeFired, StatMtxFired) - , fired_suppressions(8) { + , fired_suppressions(8) + , clock_alloc("clock allocator") { } // The objects are allocated in TLS, so one may rely on zero-initialization. @@ -381,7 +382,7 @@ void Initialize(ThreadState *thr) { // Initialize thread 0. int tid = ThreadCreate(thr, 0, 0, true); CHECK_EQ(tid, 0); - ThreadStart(thr, tid, internal_getpid()); + ThreadStart(thr, tid, GetTid(), /*workerthread*/ false); #if TSAN_CONTAINS_UBSAN __ubsan::InitAsPlugin(); #endif @@ -866,9 +867,8 @@ static void MemoryRangeSet(ThreadState *thr, uptr pc, uptr addr, uptr size, // Don't want to touch lots of shadow memory. // If a program maps 10MB stack, there is no need reset the whole range. size = (size + (kShadowCell - 1)) & ~(kShadowCell - 1); - // UnmapOrDie/MmapFixedNoReserve does not work on Windows, - // so we do it only for C/C++. - if (SANITIZER_GO || size < common_flags()->clear_shadow_mmap_threshold) { + // UnmapOrDie/MmapFixedNoReserve does not work on Windows. + if (SANITIZER_WINDOWS || size < common_flags()->clear_shadow_mmap_threshold) { u64 *p = (u64*)MemToShadow(addr); CHECK(IsShadowMem((uptr)p)); CHECK(IsShadowMem((uptr)(p + size * kShadowCnt / kShadowCell - 1))); @@ -980,21 +980,21 @@ void FuncExit(ThreadState *thr) { thr->shadow_stack_pos--; } -void ThreadIgnoreBegin(ThreadState *thr, uptr pc) { +void ThreadIgnoreBegin(ThreadState *thr, uptr pc, bool save_stack) { DPrintf("#%d: ThreadIgnoreBegin\n", thr->tid); thr->ignore_reads_and_writes++; CHECK_GT(thr->ignore_reads_and_writes, 0); thr->fast_state.SetIgnoreBit(); #if !SANITIZER_GO - if (!ctx->after_multithreaded_fork) + if (save_stack && !ctx->after_multithreaded_fork) thr->mop_ignore_set.Add(CurrentStackId(thr, pc)); #endif } void ThreadIgnoreEnd(ThreadState *thr, uptr pc) { DPrintf("#%d: ThreadIgnoreEnd\n", thr->tid); + CHECK_GT(thr->ignore_reads_and_writes, 0); thr->ignore_reads_and_writes--; - CHECK_GE(thr->ignore_reads_and_writes, 0); if (thr->ignore_reads_and_writes == 0) { thr->fast_state.ClearIgnoreBit(); #if !SANITIZER_GO @@ -1011,20 +1011,20 @@ uptr __tsan_testonly_shadow_stack_current_size() { } #endif -void ThreadIgnoreSyncBegin(ThreadState *thr, uptr pc) { +void ThreadIgnoreSyncBegin(ThreadState *thr, uptr pc, bool save_stack) { DPrintf("#%d: ThreadIgnoreSyncBegin\n", thr->tid); thr->ignore_sync++; CHECK_GT(thr->ignore_sync, 0); #if !SANITIZER_GO - if (!ctx->after_multithreaded_fork) + if (save_stack && !ctx->after_multithreaded_fork) thr->sync_ignore_set.Add(CurrentStackId(thr, pc)); #endif } void ThreadIgnoreSyncEnd(ThreadState *thr, uptr pc) { DPrintf("#%d: ThreadIgnoreSyncEnd\n", thr->tid); + CHECK_GT(thr->ignore_sync, 0); thr->ignore_sync--; - CHECK_GE(thr->ignore_sync, 0); #if !SANITIZER_GO if (thr->ignore_sync == 0) thr->sync_ignore_set.Reset(); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.h index 7fcb9d4..2cf2e16 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.h @@ -55,16 +55,22 @@ namespace __tsan { #if !SANITIZER_GO struct MapUnmapCallback; #if defined(__mips64) || defined(__aarch64__) || defined(__powerpc__) -static const uptr kAllocatorSpace = 0; -static const uptr kAllocatorSize = SANITIZER_MMAP_RANGE_SIZE; static const uptr kAllocatorRegionSizeLog = 20; static const uptr kAllocatorNumRegions = - kAllocatorSize >> kAllocatorRegionSizeLog; + SANITIZER_MMAP_RANGE_SIZE >> kAllocatorRegionSizeLog; typedef TwoLevelByteMap<(kAllocatorNumRegions >> 12), 1 << 12, MapUnmapCallback> ByteMap; -typedef SizeClassAllocator32<kAllocatorSpace, kAllocatorSize, 0, - CompactSizeClassMap, kAllocatorRegionSizeLog, ByteMap, - MapUnmapCallback> PrimaryAllocator; +struct AP32 { + static const uptr kSpaceBeg = 0; + static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE; + static const uptr kMetadataSize = 0; + typedef __sanitizer::CompactSizeClassMap SizeClassMap; + static const uptr kRegionSizeLog = kAllocatorRegionSizeLog; + typedef __tsan::ByteMap ByteMap; + typedef __tsan::MapUnmapCallback MapUnmapCallback; + static const uptr kFlags = 0; +}; +typedef SizeClassAllocator32<AP32> PrimaryAllocator; #else struct AP64 { // Allocator64 parameters. Deliberately using a short name. static const uptr kSpaceBeg = Mapping::kHeapMemBeg; @@ -381,6 +387,7 @@ struct ThreadState { // for better performance. int ignore_reads_and_writes; int ignore_sync; + int suppress_reports; // Go does not support ignores. #if !SANITIZER_GO IgnoreSet mop_ignore_set; @@ -545,6 +552,10 @@ struct Context { extern Context *ctx; // The one and the only global runtime context. +ALWAYS_INLINE Flags *flags() { + return &ctx->flags; +} + struct ScopedIgnoreInterceptors { ScopedIgnoreInterceptors() { #if !SANITIZER_GO @@ -559,12 +570,16 @@ struct ScopedIgnoreInterceptors { } }; +const char *GetObjectTypeFromTag(uptr tag); +const char *GetReportHeaderFromTag(uptr tag); +uptr TagFromShadowStackFrame(uptr pc); + class ScopedReport { public: - explicit ScopedReport(ReportType typ); + explicit ScopedReport(ReportType typ, uptr tag = kExternalTagNone); ~ScopedReport(); - void AddMemoryAccess(uptr addr, Shadow s, StackTrace stack, + void AddMemoryAccess(uptr addr, uptr external_tag, Shadow s, StackTrace stack, const MutexSet *mset); void AddStack(StackTrace stack, bool suppressable = false); void AddThread(const ThreadContext *tctx, bool suppressable = false); @@ -592,10 +607,26 @@ class ScopedReport { ThreadContext *IsThreadStackOrTls(uptr addr, bool *is_stack); void RestoreStack(int tid, const u64 epoch, VarSizeStackTrace *stk, - MutexSet *mset); + MutexSet *mset, uptr *tag = nullptr); + +// The stack could look like: +// <start> | <main> | <foo> | tag | <bar> +// This will extract the tag and keep: +// <start> | <main> | <foo> | <bar> +template<typename StackTraceTy> +void ExtractTagFromStack(StackTraceTy *stack, uptr *tag = nullptr) { + if (stack->size < 2) return; + uptr possible_tag_pc = stack->trace[stack->size - 2]; + uptr possible_tag = TagFromShadowStackFrame(possible_tag_pc); + if (possible_tag == kExternalTagNone) return; + stack->trace_buffer[stack->size - 2] = stack->trace_buffer[stack->size - 1]; + stack->size -= 1; + if (tag) *tag = possible_tag; +} template<typename StackTraceTy> -void ObtainCurrentStack(ThreadState *thr, uptr toppc, StackTraceTy *stack) { +void ObtainCurrentStack(ThreadState *thr, uptr toppc, StackTraceTy *stack, + uptr *tag = nullptr) { uptr size = thr->shadow_stack_pos - thr->shadow_stack; uptr start = 0; if (size + !!toppc > kStackTraceMax) { @@ -603,6 +634,7 @@ void ObtainCurrentStack(ThreadState *thr, uptr toppc, StackTraceTy *stack) { size = kStackTraceMax - !!toppc; } stack->Init(&thr->shadow_stack[start], size, toppc); + ExtractTagFromStack(stack, tag); } @@ -704,16 +736,16 @@ void MemoryResetRange(ThreadState *thr, uptr pc, uptr addr, uptr size); void MemoryRangeFreed(ThreadState *thr, uptr pc, uptr addr, uptr size); void MemoryRangeImitateWrite(ThreadState *thr, uptr pc, uptr addr, uptr size); -void ThreadIgnoreBegin(ThreadState *thr, uptr pc); +void ThreadIgnoreBegin(ThreadState *thr, uptr pc, bool save_stack = true); void ThreadIgnoreEnd(ThreadState *thr, uptr pc); -void ThreadIgnoreSyncBegin(ThreadState *thr, uptr pc); +void ThreadIgnoreSyncBegin(ThreadState *thr, uptr pc, bool save_stack = true); void ThreadIgnoreSyncEnd(ThreadState *thr, uptr pc); void FuncEntry(ThreadState *thr, uptr pc); void FuncExit(ThreadState *thr); int ThreadCreate(ThreadState *thr, uptr pc, uptr uid, bool detached); -void ThreadStart(ThreadState *thr, int tid, uptr os_id); +void ThreadStart(ThreadState *thr, int tid, tid_t os_id, bool workerthread); void ThreadFinish(ThreadState *thr); int ThreadTid(ThreadState *thr, uptr pc, uptr uid); void ThreadJoin(ThreadState *thr, uptr pc, int tid); @@ -728,13 +760,16 @@ void ProcDestroy(Processor *proc); void ProcWire(Processor *proc, ThreadState *thr); void ProcUnwire(Processor *proc, ThreadState *thr); -void MutexCreate(ThreadState *thr, uptr pc, uptr addr, - bool rw, bool recursive, bool linker_init); -void MutexDestroy(ThreadState *thr, uptr pc, uptr addr); -void MutexLock(ThreadState *thr, uptr pc, uptr addr, int rec = 1, - bool try_lock = false); -int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, bool all = false); -void MutexReadLock(ThreadState *thr, uptr pc, uptr addr, bool try_lock = false); +// Note: the parameter is called flagz, because flags is already taken +// by the global function that returns flags. +void MutexCreate(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); +void MutexDestroy(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); +void MutexPreLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); +void MutexPostLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0, + int rec = 1); +int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); +void MutexPreReadLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); +void MutexPostReadLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz = 0); void MutexReadUnlock(ThreadState *thr, uptr pc, uptr addr); void MutexReadOrWriteUnlock(ThreadState *thr, uptr pc, uptr addr); void MutexRepair(ThreadState *thr, uptr pc, uptr addr); // call on EOWNERDEAD diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_aarch64.S b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_aarch64.S index ef06f04..61171d6 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_aarch64.S +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_aarch64.S @@ -1,13 +1,46 @@ +// The content of this file is AArch64-only: +#if defined(__aarch64__) + #include "sanitizer_common/sanitizer_asm.h" +#if !defined(__APPLE__) .section .bss .type __tsan_pointer_chk_guard, %object -.size __tsan_pointer_chk_guard, 8 +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(__tsan_pointer_chk_guard)) __tsan_pointer_chk_guard: .zero 8 +#endif + +#if defined(__APPLE__) +.align 2 + +.section __DATA,__nl_symbol_ptr,non_lazy_symbol_pointers +.long _setjmp$non_lazy_ptr +_setjmp$non_lazy_ptr: +.indirect_symbol _setjmp +.long 0 + +.section __DATA,__nl_symbol_ptr,non_lazy_symbol_pointers +.long __setjmp$non_lazy_ptr +__setjmp$non_lazy_ptr: +.indirect_symbol __setjmp +.long 0 + +.section __DATA,__nl_symbol_ptr,non_lazy_symbol_pointers +.long _sigsetjmp$non_lazy_ptr +_sigsetjmp$non_lazy_ptr: +.indirect_symbol _sigsetjmp +.long 0 +#endif +#if !defined(__APPLE__) .section .text +#else +.section __TEXT,__text +.align 3 +#endif +#if !defined(__APPLE__) // GLIBC mangles the function pointers in jmp_buf (used in {set,long}*jmp // functions) by XORing them with a random guard pointer. For AArch64 it is a // global variable rather than a TCB one (as for x86_64/powerpc) and althought @@ -16,9 +49,9 @@ __tsan_pointer_chk_guard: // not stable). So InitializeGuardPtr obtains the pointer guard value by // issuing a setjmp and checking the resulting pointers values against the // original ones. -.hidden _Z18InitializeGuardPtrv +ASM_HIDDEN(_Z18InitializeGuardPtrv) .global _Z18InitializeGuardPtrv -.type _Z18InitializeGuardPtrv, @function +ASM_TYPE_FUNCTION(ASM_TSAN_SYMBOL_INTERCEPTOR(_Z18InitializeGuardPtrv)) _Z18InitializeGuardPtrv: CFI_STARTPROC // Allocates a jmp_buf for the setjmp call. @@ -55,12 +88,14 @@ _Z18InitializeGuardPtrv: CFI_DEF_CFA (31, 0) ret CFI_ENDPROC -.size _Z18InitializeGuardPtrv, .-_Z18InitializeGuardPtrv +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(_Z18InitializeGuardPtrv)) +#endif -.hidden __tsan_setjmp +ASM_HIDDEN(__tsan_setjmp) .comm _ZN14__interception11real_setjmpE,8,8 -.type setjmp, @function -setjmp: +.globl ASM_TSAN_SYMBOL_INTERCEPTOR(setjmp) +ASM_TYPE_FUNCTION(ASM_TSAN_SYMBOL_INTERCEPTOR(setjmp)) +ASM_TSAN_SYMBOL_INTERCEPTOR(setjmp): CFI_STARTPROC // save env parameters for function call @@ -78,14 +113,19 @@ setjmp: CFI_OFFSET (19, -16) mov x19, x0 +#if !defined(__APPLE__) // SP pointer mangling (see glibc setjmp) adrp x2, __tsan_pointer_chk_guard ldr x2, [x2, #:lo12:__tsan_pointer_chk_guard] add x0, x29, 32 eor x1, x2, x0 +#else + add x0, x29, 32 + mov x1, x0 +#endif // call tsan interceptor - bl __tsan_setjmp + bl ASM_TSAN_SYMBOL(__tsan_setjmp) // restore env parameter mov x0, x19 @@ -96,18 +136,24 @@ setjmp: CFI_DEF_CFA (31, 0) // tail jump to libc setjmp +#if !defined(__APPLE__) adrp x1, :got:_ZN14__interception11real_setjmpE ldr x1, [x1, #:got_lo12:_ZN14__interception11real_setjmpE] ldr x1, [x1] +#else + adrp x1, _setjmp$non_lazy_ptr@page + add x1, x1, _setjmp$non_lazy_ptr@pageoff + ldr x1, [x1] +#endif br x1 CFI_ENDPROC -.size setjmp, .-setjmp +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(setjmp)) .comm _ZN14__interception12real__setjmpE,8,8 -.globl _setjmp -.type _setjmp, @function -_setjmp: +.globl ASM_TSAN_SYMBOL_INTERCEPTOR(_setjmp) +ASM_TYPE_FUNCTION(ASM_TSAN_SYMBOL_INTERCEPTOR(_setjmp)) +ASM_TSAN_SYMBOL_INTERCEPTOR(_setjmp): CFI_STARTPROC // save env parameters for function call @@ -125,14 +171,19 @@ _setjmp: CFI_OFFSET (19, -16) mov x19, x0 +#if !defined(__APPLE__) // SP pointer mangling (see glibc setjmp) adrp x2, __tsan_pointer_chk_guard ldr x2, [x2, #:lo12:__tsan_pointer_chk_guard] add x0, x29, 32 eor x1, x2, x0 +#else + add x0, x29, 32 + mov x1, x0 +#endif // call tsan interceptor - bl __tsan_setjmp + bl ASM_TSAN_SYMBOL(__tsan_setjmp) // Restore jmp_buf parameter mov x0, x19 @@ -143,18 +194,24 @@ _setjmp: CFI_DEF_CFA (31, 0) // tail jump to libc setjmp +#if !defined(__APPLE__) adrp x1, :got:_ZN14__interception12real__setjmpE ldr x1, [x1, #:got_lo12:_ZN14__interception12real__setjmpE] ldr x1, [x1] +#else + adrp x1, __setjmp$non_lazy_ptr@page + add x1, x1, __setjmp$non_lazy_ptr@pageoff + ldr x1, [x1] +#endif br x1 CFI_ENDPROC -.size _setjmp, .-_setjmp +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(_setjmp)) .comm _ZN14__interception14real_sigsetjmpE,8,8 -.globl sigsetjmp -.type sigsetjmp, @function -sigsetjmp: +.globl ASM_TSAN_SYMBOL_INTERCEPTOR(sigsetjmp) +ASM_TYPE_FUNCTION(ASM_TSAN_SYMBOL_INTERCEPTOR(sigsetjmp)) +ASM_TSAN_SYMBOL_INTERCEPTOR(sigsetjmp): CFI_STARTPROC // save env parameters for function call @@ -174,14 +231,19 @@ sigsetjmp: mov w20, w1 mov x19, x0 +#if !defined(__APPLE__) // SP pointer mangling (see glibc setjmp) adrp x2, __tsan_pointer_chk_guard ldr x2, [x2, #:lo12:__tsan_pointer_chk_guard] add x0, x29, 32 eor x1, x2, x0 +#else + add x0, x29, 32 + mov x1, x0 +#endif // call tsan interceptor - bl __tsan_setjmp + bl ASM_TSAN_SYMBOL(__tsan_setjmp) // restore env parameter mov w1, w20 @@ -195,17 +257,24 @@ sigsetjmp: CFI_DEF_CFA (31, 0) // tail jump to libc sigsetjmp +#if !defined(__APPLE__) adrp x2, :got:_ZN14__interception14real_sigsetjmpE ldr x2, [x2, #:got_lo12:_ZN14__interception14real_sigsetjmpE] ldr x2, [x2] +#else + adrp x2, _sigsetjmp$non_lazy_ptr@page + add x2, x2, _sigsetjmp$non_lazy_ptr@pageoff + ldr x2, [x2] +#endif br x2 CFI_ENDPROC -.size sigsetjmp, .-sigsetjmp +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(sigsetjmp)) +#if !defined(__APPLE__) .comm _ZN14__interception16real___sigsetjmpE,8,8 -.globl __sigsetjmp -.type __sigsetjmp, @function -__sigsetjmp: +.globl ASM_TSAN_SYMBOL_INTERCEPTOR(__sigsetjmp) +ASM_TYPE_FUNCTION(ASM_TSAN_SYMBOL_INTERCEPTOR(__sigsetjmp)) +ASM_TSAN_SYMBOL_INTERCEPTOR(__sigsetjmp): CFI_STARTPROC // save env parameters for function call @@ -225,14 +294,16 @@ __sigsetjmp: mov w20, w1 mov x19, x0 +#if !defined(__APPLE__) // SP pointer mangling (see glibc setjmp) adrp x2, __tsan_pointer_chk_guard ldr x2, [x2, #:lo12:__tsan_pointer_chk_guard] add x0, x29, 32 eor x1, x2, x0 +#endif // call tsan interceptor - bl __tsan_setjmp + bl ASM_TSAN_SYMBOL(__tsan_setjmp) mov w1, w20 mov x0, x19 @@ -245,14 +316,22 @@ __sigsetjmp: CFI_DEF_CFA (31, 0) // tail jump to libc __sigsetjmp +#if !defined(__APPLE__) adrp x2, :got:_ZN14__interception16real___sigsetjmpE ldr x2, [x2, #:got_lo12:_ZN14__interception16real___sigsetjmpE] ldr x2, [x2] +#else + adrp x2, ASM_TSAN_SYMBOL(__sigsetjmp)@page + add x2, x2, ASM_TSAN_SYMBOL(__sigsetjmp)@pageoff +#endif br x2 CFI_ENDPROC -.size __sigsetjmp, .-__sigsetjmp +ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(__sigsetjmp)) +#endif #if defined(__linux__) /* We do not need executable stack. */ .section .note.GNU-stack,"",@progbits #endif + +#endif diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S index caa8323..98947fd 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S @@ -1,4 +1,8 @@ +// The content of this file is x86_64-only: +#if defined(__x86_64__) + #include "sanitizer_common/sanitizer_asm.h" + #if !defined(__APPLE__) .section .text #else @@ -357,3 +361,5 @@ ASM_SIZE(ASM_TSAN_SYMBOL_INTERCEPTOR(__sigsetjmp)) /* We do not need executable stack. */ .section .note.GNU-stack,"",@progbits #endif + +#endif diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cc index f3b51c3..2f85811 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cc @@ -62,32 +62,29 @@ static void ReportMutexMisuse(ThreadState *thr, uptr pc, ReportType typ, OutputReport(thr, rep); } -void MutexCreate(ThreadState *thr, uptr pc, uptr addr, - bool rw, bool recursive, bool linker_init) { - DPrintf("#%d: MutexCreate %zx\n", thr->tid, addr); +void MutexCreate(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { + DPrintf("#%d: MutexCreate %zx flagz=0x%x\n", thr->tid, addr, flagz); StatInc(thr, StatMutexCreate); - if (!linker_init && IsAppMem(addr)) { + if (!(flagz & MutexFlagLinkerInit) && IsAppMem(addr)) { CHECK(!thr->is_freeing); thr->is_freeing = true; MemoryWrite(thr, pc, addr, kSizeLog1); thr->is_freeing = false; } SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, true); - s->is_rw = rw; - s->is_recursive = recursive; - s->is_linker_init = linker_init; + s->SetFlags(flagz & MutexCreationFlagMask); if (!SANITIZER_GO && s->creation_stack_id == 0) s->creation_stack_id = CurrentStackId(thr, pc); s->mtx.Unlock(); } -void MutexDestroy(ThreadState *thr, uptr pc, uptr addr) { +void MutexDestroy(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { DPrintf("#%d: MutexDestroy %zx\n", thr->tid, addr); StatInc(thr, StatMutexDestroy); SyncVar *s = ctx->metamap.GetIfExistsAndLock(addr, true); if (s == 0) return; - if (s->is_linker_init) { + if ((flagz & MutexFlagLinkerInit) || s->IsFlagSet(MutexFlagLinkerInit)) { // Destroy is no-op for linker-initialized mutexes. s->mtx.Unlock(); return; @@ -100,8 +97,8 @@ void MutexDestroy(ThreadState *thr, uptr pc, uptr addr) { bool unlock_locked = false; if (flags()->report_destroy_locked && s->owner_tid != SyncVar::kInvalidTid - && !s->is_broken) { - s->is_broken = true; + && !s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); unlock_locked = true; } u64 mid = s->GetId(); @@ -141,12 +138,33 @@ void MutexDestroy(ThreadState *thr, uptr pc, uptr addr) { // s will be destroyed and freed in MetaMap::FreeBlock. } -void MutexLock(ThreadState *thr, uptr pc, uptr addr, int rec, bool try_lock) { - DPrintf("#%d: MutexLock %zx rec=%d\n", thr->tid, addr, rec); - CHECK_GT(rec, 0); +void MutexPreLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { + DPrintf("#%d: MutexPreLock %zx flagz=0x%x\n", thr->tid, addr, flagz); + if (!(flagz & MutexFlagTryLock) && common_flags()->detect_deadlocks) { + SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, false); + s->UpdateFlags(flagz); + if (s->owner_tid != thr->tid) { + Callback cb(thr, pc); + ctx->dd->MutexBeforeLock(&cb, &s->dd, true); + s->mtx.ReadUnlock(); + ReportDeadlock(thr, pc, ctx->dd->GetReport(&cb)); + } else { + s->mtx.ReadUnlock(); + } + } +} + +void MutexPostLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz, int rec) { + DPrintf("#%d: MutexPostLock %zx flag=0x%x rec=%d\n", + thr->tid, addr, flagz, rec); + if (flagz & MutexFlagRecursiveLock) + CHECK_GT(rec, 0); + else + rec = 1; if (IsAppMem(addr)) MemoryReadAtomic(thr, pc, addr, kSizeLog1); SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, true); + s->UpdateFlags(flagz); thr->fast_state.IncrementEpoch(); TraceAddEvent(thr, thr->fast_state, EventTypeLock, s->GetId()); bool report_double_lock = false; @@ -156,38 +174,43 @@ void MutexLock(ThreadState *thr, uptr pc, uptr addr, int rec, bool try_lock) { s->last_lock = thr->fast_state.raw(); } else if (s->owner_tid == thr->tid) { CHECK_GT(s->recursion, 0); - } else if (flags()->report_mutex_bugs && !s->is_broken) { - s->is_broken = true; + } else if (flags()->report_mutex_bugs && !s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); report_double_lock = true; } - if (s->recursion == 0) { + const bool first = s->recursion == 0; + s->recursion += rec; + if (first) { StatInc(thr, StatMutexLock); AcquireImpl(thr, pc, &s->clock); AcquireImpl(thr, pc, &s->read_clock); - } else if (!s->is_recursive) { + } else if (!s->IsFlagSet(MutexFlagWriteReentrant)) { StatInc(thr, StatMutexRecLock); } - s->recursion += rec; thr->mset.Add(s->GetId(), true, thr->fast_state.epoch()); - if (common_flags()->detect_deadlocks && (s->recursion - rec) == 0) { + bool pre_lock = false; + if (first && common_flags()->detect_deadlocks) { + pre_lock = (flagz & MutexFlagDoPreLockOnPostLock) && + !(flagz & MutexFlagTryLock); Callback cb(thr, pc); - if (!try_lock) + if (pre_lock) ctx->dd->MutexBeforeLock(&cb, &s->dd, true); - ctx->dd->MutexAfterLock(&cb, &s->dd, true, try_lock); + ctx->dd->MutexAfterLock(&cb, &s->dd, true, flagz & MutexFlagTryLock); } u64 mid = s->GetId(); s->mtx.Unlock(); // Can't touch s after this point. + s = 0; if (report_double_lock) ReportMutexMisuse(thr, pc, ReportTypeMutexDoubleLock, addr, mid); - if (common_flags()->detect_deadlocks) { + if (first && pre_lock && common_flags()->detect_deadlocks) { Callback cb(thr, pc); ReportDeadlock(thr, pc, ctx->dd->GetReport(&cb)); } } -int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, bool all) { - DPrintf("#%d: MutexUnlock %zx all=%d\n", thr->tid, addr, all); +int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { + DPrintf("#%d: MutexUnlock %zx flagz=0x%x\n", thr->tid, addr, flagz); if (IsAppMem(addr)) MemoryReadAtomic(thr, pc, addr, kSizeLog1); SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, true); @@ -196,12 +219,12 @@ int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, bool all) { int rec = 0; bool report_bad_unlock = false; if (!SANITIZER_GO && (s->recursion == 0 || s->owner_tid != thr->tid)) { - if (flags()->report_mutex_bugs && !s->is_broken) { - s->is_broken = true; + if (flags()->report_mutex_bugs && !s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); report_bad_unlock = true; } } else { - rec = all ? s->recursion : 1; + rec = (flagz & MutexFlagRecursiveUnlock) ? s->recursion : 1; s->recursion -= rec; if (s->recursion == 0) { StatInc(thr, StatMutexUnlock); @@ -229,36 +252,53 @@ int MutexUnlock(ThreadState *thr, uptr pc, uptr addr, bool all) { return rec; } -void MutexReadLock(ThreadState *thr, uptr pc, uptr addr, bool trylock) { - DPrintf("#%d: MutexReadLock %zx\n", thr->tid, addr); +void MutexPreReadLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { + DPrintf("#%d: MutexPreReadLock %zx flagz=0x%x\n", thr->tid, addr, flagz); + if (!(flagz & MutexFlagTryLock) && common_flags()->detect_deadlocks) { + SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, false); + s->UpdateFlags(flagz); + Callback cb(thr, pc); + ctx->dd->MutexBeforeLock(&cb, &s->dd, false); + s->mtx.ReadUnlock(); + ReportDeadlock(thr, pc, ctx->dd->GetReport(&cb)); + } +} + +void MutexPostReadLock(ThreadState *thr, uptr pc, uptr addr, u32 flagz) { + DPrintf("#%d: MutexPostReadLock %zx flagz=0x%x\n", thr->tid, addr, flagz); StatInc(thr, StatMutexReadLock); if (IsAppMem(addr)) MemoryReadAtomic(thr, pc, addr, kSizeLog1); SyncVar *s = ctx->metamap.GetOrCreateAndLock(thr, pc, addr, false); + s->UpdateFlags(flagz); thr->fast_state.IncrementEpoch(); TraceAddEvent(thr, thr->fast_state, EventTypeRLock, s->GetId()); bool report_bad_lock = false; if (s->owner_tid != SyncVar::kInvalidTid) { - if (flags()->report_mutex_bugs && !s->is_broken) { - s->is_broken = true; + if (flags()->report_mutex_bugs && !s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); report_bad_lock = true; } } AcquireImpl(thr, pc, &s->clock); s->last_lock = thr->fast_state.raw(); thr->mset.Add(s->GetId(), false, thr->fast_state.epoch()); - if (common_flags()->detect_deadlocks && s->recursion == 0) { + bool pre_lock = false; + if (common_flags()->detect_deadlocks) { + pre_lock = (flagz & MutexFlagDoPreLockOnPostLock) && + !(flagz & MutexFlagTryLock); Callback cb(thr, pc); - if (!trylock) + if (pre_lock) ctx->dd->MutexBeforeLock(&cb, &s->dd, false); - ctx->dd->MutexAfterLock(&cb, &s->dd, false, trylock); + ctx->dd->MutexAfterLock(&cb, &s->dd, false, flagz & MutexFlagTryLock); } u64 mid = s->GetId(); s->mtx.ReadUnlock(); // Can't touch s after this point. + s = 0; if (report_bad_lock) ReportMutexMisuse(thr, pc, ReportTypeMutexBadReadLock, addr, mid); - if (common_flags()->detect_deadlocks) { + if (pre_lock && common_flags()->detect_deadlocks) { Callback cb(thr, pc); ReportDeadlock(thr, pc, ctx->dd->GetReport(&cb)); } @@ -274,8 +314,8 @@ void MutexReadUnlock(ThreadState *thr, uptr pc, uptr addr) { TraceAddEvent(thr, thr->fast_state, EventTypeRUnlock, s->GetId()); bool report_bad_unlock = false; if (s->owner_tid != SyncVar::kInvalidTid) { - if (flags()->report_mutex_bugs && !s->is_broken) { - s->is_broken = true; + if (flags()->report_mutex_bugs && !s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); report_bad_unlock = true; } } @@ -323,8 +363,8 @@ void MutexReadOrWriteUnlock(ThreadState *thr, uptr pc, uptr addr) { } else { StatInc(thr, StatMutexRecUnlock); } - } else if (!s->is_broken) { - s->is_broken = true; + } else if (!s->IsFlagSet(MutexFlagBroken)) { + s->SetFlags(MutexFlagBroken); report_bad_unlock = true; } thr->mset.Del(s->GetId(), write); @@ -373,10 +413,10 @@ void Acquire(ThreadState *thr, uptr pc, uptr addr) { static void UpdateClockCallback(ThreadContextBase *tctx_base, void *arg) { ThreadState *thr = reinterpret_cast<ThreadState*>(arg); ThreadContext *tctx = static_cast<ThreadContext*>(tctx_base); + u64 epoch = tctx->epoch1; if (tctx->status == ThreadStatusRunning) - thr->clock.set(tctx->tid, tctx->thr->fast_state.epoch()); - else - thr->clock.set(tctx->tid, tctx->epoch1); + epoch = tctx->thr->fast_state.epoch(); + thr->clock.set(&thr->proc()->clock_cache, tctx->tid, epoch); } void AcquireGlobal(ThreadState *thr, uptr pc) { @@ -416,10 +456,10 @@ void ReleaseStore(ThreadState *thr, uptr pc, uptr addr) { static void UpdateSleepClockCallback(ThreadContextBase *tctx_base, void *arg) { ThreadState *thr = reinterpret_cast<ThreadState*>(arg); ThreadContext *tctx = static_cast<ThreadContext*>(tctx_base); + u64 epoch = tctx->epoch1; if (tctx->status == ThreadStatusRunning) - thr->last_sleep_clock.set(tctx->tid, tctx->thr->fast_state.epoch()); - else - thr->last_sleep_clock.set(tctx->tid, tctx->epoch1); + epoch = tctx->thr->fast_state.epoch(); + thr->last_sleep_clock.set(&thr->proc()->clock_cache, tctx->tid, epoch); } void AfterSleep(ThreadState *thr, uptr pc) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc index bc8944f..85a9829 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc @@ -143,11 +143,12 @@ static ReportStack *SymbolizeStack(StackTrace trace) { return stack; } -ScopedReport::ScopedReport(ReportType typ) { +ScopedReport::ScopedReport(ReportType typ, uptr tag) { ctx->thread_registry->CheckLocked(); void *mem = internal_alloc(MBlockReport, sizeof(ReportDesc)); rep_ = new(mem) ReportDesc; rep_->typ = typ; + rep_->tag = tag; ctx->report_mtx.Lock(); CommonSanitizerReportMutex.Lock(); } @@ -164,8 +165,8 @@ void ScopedReport::AddStack(StackTrace stack, bool suppressable) { (*rs)->suppressable = suppressable; } -void ScopedReport::AddMemoryAccess(uptr addr, Shadow s, StackTrace stack, - const MutexSet *mset) { +void ScopedReport::AddMemoryAccess(uptr addr, uptr external_tag, Shadow s, + StackTrace stack, const MutexSet *mset) { void *mem = internal_alloc(MBlockReportMop, sizeof(ReportMop)); ReportMop *mop = new(mem) ReportMop; rep_->mops.PushBack(mop); @@ -175,6 +176,7 @@ void ScopedReport::AddMemoryAccess(uptr addr, Shadow s, StackTrace stack, mop->write = s.IsWrite(); mop->atomic = s.IsAtomic(); mop->stack = SymbolizeStack(stack); + mop->external_tag = external_tag; if (mop->stack) mop->stack->suppressable = true; for (uptr i = 0; i < mset->Size(); i++) { @@ -202,6 +204,7 @@ void ScopedReport::AddThread(const ThreadContext *tctx, bool suppressable) { rt->running = (tctx->status == ThreadStatusRunning); rt->name = internal_strdup(tctx->name); rt->parent_tid = tctx->parent_tid; + rt->workerthread = tctx->workerthread; rt->stack = 0; rt->stack = SymbolizeStackId(tctx->creation_stack_id); if (rt->stack) @@ -311,7 +314,7 @@ void ScopedReport::AddLocation(uptr addr, uptr size) { return; #if !SANITIZER_GO int fd = -1; - int creat_tid = -1; + int creat_tid = kInvalidTid; u32 creat_stack = 0; if (FdLocation(addr, &fd, &creat_tid, &creat_stack)) { ReportLocation *loc = ReportLocation::New(ReportLocationFD); @@ -336,6 +339,7 @@ void ScopedReport::AddLocation(uptr addr, uptr size) { ReportLocation *loc = ReportLocation::New(ReportLocationHeap); loc->heap_chunk_start = (uptr)allocator()->GetBlockBegin((void *)addr); loc->heap_chunk_size = b->siz; + loc->external_tag = b->tag; loc->tid = tctx ? tctx->tid : b->tid; loc->stack = SymbolizeStackId(b->stk); rep_->locs.PushBack(loc); @@ -374,7 +378,7 @@ const ReportDesc *ScopedReport::GetReport() const { } void RestoreStack(int tid, const u64 epoch, VarSizeStackTrace *stk, - MutexSet *mset) { + MutexSet *mset, uptr *tag) { // This function restores stack trace and mutex set for the thread/epoch. // It does so by getting stack trace and mutex set at the beginning of // trace part, and then replaying the trace till the given epoch. @@ -433,6 +437,7 @@ void RestoreStack(int tid, const u64 epoch, VarSizeStackTrace *stk, return; pos++; stk->Init(&stack[0], pos); + ExtractTagFromStack(stk, tag); } static bool HandleRacyStacks(ThreadState *thr, VarSizeStackTrace traces[2], @@ -497,7 +502,7 @@ static void AddRacyStacks(ThreadState *thr, VarSizeStackTrace traces[2], } bool OutputReport(ThreadState *thr, const ScopedReport &srep) { - if (!flags()->report_bugs) + if (!flags()->report_bugs || thr->suppress_reports) return false; atomic_store_relaxed(&ctx->last_symbolize_time_ns, NanoTime()); const ReportDesc *rep = srep.GetReport(); @@ -628,8 +633,9 @@ void ReportRace(ThreadState *thr) { const uptr kMop = 2; VarSizeStackTrace traces[kMop]; + uptr tags[kMop] = {kExternalTagNone}; const uptr toppc = TraceTopPC(thr); - ObtainCurrentStack(thr, toppc, &traces[0]); + ObtainCurrentStack(thr, toppc, &traces[0], &tags[0]); if (IsFiredSuppression(ctx, typ, traces[0])) return; @@ -639,18 +645,29 @@ void ReportRace(ThreadState *thr) { MutexSet *mset2 = new(&mset_buffer[0]) MutexSet(); Shadow s2(thr->racy_state[1]); - RestoreStack(s2.tid(), s2.epoch(), &traces[1], mset2); + RestoreStack(s2.tid(), s2.epoch(), &traces[1], mset2, &tags[1]); if (IsFiredSuppression(ctx, typ, traces[1])) return; if (HandleRacyStacks(thr, traces, addr_min, addr_max)) return; + // If any of the accesses has a tag, treat this as an "external" race. + uptr tag = kExternalTagNone; + for (uptr i = 0; i < kMop; i++) { + if (tags[i] != kExternalTagNone) { + typ = ReportTypeExternalRace; + tag = tags[i]; + break; + } + } + ThreadRegistryLock l0(ctx->thread_registry); - ScopedReport rep(typ); + ScopedReport rep(typ, tag); for (uptr i = 0; i < kMop; i++) { Shadow s(thr->racy_state[i]); - rep.AddMemoryAccess(addr, s, traces[i], i == 0 ? &thr->mset : mset2); + rep.AddMemoryAccess(addr, tags[i], s, traces[i], + i == 0 ? &thr->mset : mset2); } for (uptr i = 0; i < kMop; i++) { diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_thread.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_thread.cc index 5b17dc6..83fab08 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_thread.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl_thread.cc @@ -142,6 +142,10 @@ void ThreadContext::OnFinished() { if (common_flags()->detect_deadlocks) ctx->dd->DestroyLogicalThread(thr->dd_lt); + thr->clock.ResetCached(&thr->proc()->clock_cache); +#if !SANITIZER_GO + thr->last_sleep_clock.ResetCached(&thr->proc()->clock_cache); +#endif thr->~ThreadState(); #if TSAN_COLLECT_STATS StatAggregate(ctx->stat, thr->stat); @@ -236,7 +240,7 @@ int ThreadCreate(ThreadState *thr, uptr pc, uptr uid, bool detached) { return tid; } -void ThreadStart(ThreadState *thr, int tid, uptr os_id) { +void ThreadStart(ThreadState *thr, int tid, tid_t os_id, bool workerthread) { uptr stk_addr = 0; uptr stk_size = 0; uptr tls_addr = 0; @@ -248,25 +252,13 @@ void ThreadStart(ThreadState *thr, int tid, uptr os_id) { if (stk_addr && stk_size) MemoryRangeImitateWrite(thr, /*pc=*/ 1, stk_addr, stk_size); - if (tls_addr && tls_size) { - // Check that the thr object is in tls; - const uptr thr_beg = (uptr)thr; - const uptr thr_end = (uptr)thr + sizeof(*thr); - CHECK_GE(thr_beg, tls_addr); - CHECK_LE(thr_beg, tls_addr + tls_size); - CHECK_GE(thr_end, tls_addr); - CHECK_LE(thr_end, tls_addr + tls_size); - // Since the thr object is huge, skip it. - MemoryRangeImitateWrite(thr, /*pc=*/ 2, tls_addr, thr_beg - tls_addr); - MemoryRangeImitateWrite(thr, /*pc=*/ 2, - thr_end, tls_addr + tls_size - thr_end); - } + if (tls_addr && tls_size) ImitateTlsWrite(thr, tls_addr, tls_size); } #endif ThreadRegistry *tr = ctx->thread_registry; OnStartedArgs args = { thr, stk_addr, stk_size, tls_addr, tls_size }; - tr->StartThread(tid, os_id, &args); + tr->StartThread(tid, os_id, workerthread, &args); tr->Lock(); thr->tctx = (ThreadContext*)tr->GetThreadLocked(tid); @@ -357,6 +349,7 @@ void MemoryAccessRange(ThreadState *thr, uptr pc, uptr addr, StatInc(thr, StatMopRange); if (*shadow_mem == kShadowRodata) { + DCHECK(!is_write); // Access to .rodata section, no races here. // Measurements show that it can be 10-20% of all memory accesses. StatInc(thr, StatMopRangeRodata); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.cc index d1d6ed2..18c83d5 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.cc @@ -75,14 +75,11 @@ void StatOutput(u64 *stat) { name[StatClockAcquire] = "Clock acquire "; name[StatClockAcquireEmpty] = " empty clock "; name[StatClockAcquireFastRelease] = " fast from release-store "; - name[StatClockAcquireLarge] = " contains my tid "; - name[StatClockAcquireRepeat] = " repeated (fast) "; name[StatClockAcquireFull] = " full (slow) "; name[StatClockAcquiredSomething] = " acquired something "; name[StatClockRelease] = "Clock release "; name[StatClockReleaseResize] = " resize "; - name[StatClockReleaseFast1] = " fast1 "; - name[StatClockReleaseFast2] = " fast2 "; + name[StatClockReleaseFast] = " fast "; name[StatClockReleaseSlow] = " dirty overflow (slow) "; name[StatClockReleaseFull] = " full (slow) "; name[StatClockReleaseAcquired] = " was acquired "; @@ -153,6 +150,16 @@ void StatOutput(u64 *stat) { name[StatAnnotatePublishMemoryRange] = " PublishMemoryRange "; name[StatAnnotateUnpublishMemoryRange] = " UnpublishMemoryRange "; name[StatAnnotateThreadName] = " ThreadName "; + name[Stat__tsan_mutex_create] = " __tsan_mutex_create "; + name[Stat__tsan_mutex_destroy] = " __tsan_mutex_destroy "; + name[Stat__tsan_mutex_pre_lock] = " __tsan_mutex_pre_lock "; + name[Stat__tsan_mutex_post_lock] = " __tsan_mutex_post_lock "; + name[Stat__tsan_mutex_pre_unlock] = " __tsan_mutex_pre_unlock "; + name[Stat__tsan_mutex_post_unlock] = " __tsan_mutex_post_unlock "; + name[Stat__tsan_mutex_pre_signal] = " __tsan_mutex_pre_signal "; + name[Stat__tsan_mutex_post_signal] = " __tsan_mutex_post_signal "; + name[Stat__tsan_mutex_pre_divert] = " __tsan_mutex_pre_divert "; + name[Stat__tsan_mutex_post_divert] = " __tsan_mutex_post_divert "; name[StatMtxTotal] = "Contentionz "; name[StatMtxTrace] = " Trace "; diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.h index 8447dd8..42d6a2b 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_stat.h @@ -74,15 +74,12 @@ enum StatType { StatClockAcquire, StatClockAcquireEmpty, StatClockAcquireFastRelease, - StatClockAcquireLarge, - StatClockAcquireRepeat, StatClockAcquireFull, StatClockAcquiredSomething, // Clocks - release. StatClockRelease, StatClockReleaseResize, - StatClockReleaseFast1, - StatClockReleaseFast2, + StatClockReleaseFast, StatClockReleaseSlow, StatClockReleaseFull, StatClockReleaseAcquired, @@ -157,6 +154,16 @@ enum StatType { StatAnnotatePublishMemoryRange, StatAnnotateUnpublishMemoryRange, StatAnnotateThreadName, + Stat__tsan_mutex_create, + Stat__tsan_mutex_destroy, + Stat__tsan_mutex_pre_lock, + Stat__tsan_mutex_post_lock, + Stat__tsan_mutex_pre_unlock, + Stat__tsan_mutex_post_unlock, + Stat__tsan_mutex_pre_signal, + Stat__tsan_mutex_post_signal, + Stat__tsan_mutex_pre_divert, + Stat__tsan_mutex_post_divert, // Internal mutex contentionz. StatMtxTotal, diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_suppressions.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_suppressions.cc index bfb64e0..e39702b 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_suppressions.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_suppressions.cc @@ -74,6 +74,8 @@ static const char *conv(ReportType typ) { return kSuppressionRace; else if (typ == ReportTypeVptrUseAfterFree) return kSuppressionRace; + else if (typ == ReportTypeExternalRace) + return kSuppressionRace; else if (typ == ReportTypeThreadLeak) return kSuppressionThread; else if (typ == ReportTypeMutexDestroyLocked) diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.cc b/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.cc index 44c6a26..44ae558 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.cc +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.cc @@ -42,10 +42,7 @@ void SyncVar::Reset(Processor *proc) { owner_tid = kInvalidTid; last_lock = 0; recursion = 0; - is_rw = 0; - is_recursive = 0; - is_broken = 0; - is_linker_init = 0; + atomic_store_relaxed(&flags, 0); if (proc == 0) { CHECK_EQ(clock.size(), 0); @@ -56,7 +53,9 @@ void SyncVar::Reset(Processor *proc) { } } -MetaMap::MetaMap() { +MetaMap::MetaMap() + : block_alloc_("heap block allocator") + , sync_alloc_("sync allocator") { atomic_store(&uid_gen_, 0, memory_order_relaxed); } @@ -64,6 +63,7 @@ void MetaMap::AllocBlock(ThreadState *thr, uptr pc, uptr p, uptr sz) { u32 idx = block_alloc_.Alloc(&thr->proc()->block_cache); MBlock *b = block_alloc_.Map(idx); b->siz = sz; + b->tag = 0; b->tid = thr->tid; b->stk = CurrentStackId(thr, pc); u32 *meta = MemToMeta(p); diff --git a/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.h b/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.h index 86e6bbd..b83c09f 100644 --- a/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.h +++ b/contrib/compiler-rt/lib/tsan/rtl/tsan_sync.h @@ -23,6 +23,29 @@ namespace __tsan { +// These need to match __tsan_mutex_* flags defined in tsan_interface.h. +// See documentation there as well. +enum MutexFlags { + MutexFlagLinkerInit = 1 << 0, // __tsan_mutex_linker_init + MutexFlagWriteReentrant = 1 << 1, // __tsan_mutex_write_reentrant + MutexFlagReadReentrant = 1 << 2, // __tsan_mutex_read_reentrant + MutexFlagReadLock = 1 << 3, // __tsan_mutex_read_lock + MutexFlagTryLock = 1 << 4, // __tsan_mutex_try_lock + MutexFlagTryLockFailed = 1 << 5, // __tsan_mutex_try_lock_failed + MutexFlagRecursiveLock = 1 << 6, // __tsan_mutex_recursive_lock + MutexFlagRecursiveUnlock = 1 << 7, // __tsan_mutex_recursive_unlock + + // The following flags are runtime private. + // Mutex API misuse was detected, so don't report any more. + MutexFlagBroken = 1 << 30, + // We did not intercept pre lock event, so handle it on post lock. + MutexFlagDoPreLockOnPostLock = 1 << 29, + // Must list all mutex creation flags. + MutexCreationFlagMask = MutexFlagLinkerInit | + MutexFlagWriteReentrant | + MutexFlagReadReentrant, +}; + struct SyncVar { SyncVar(); @@ -35,10 +58,7 @@ struct SyncVar { int owner_tid; // Set only by exclusive owners. u64 last_lock; int recursion; - bool is_rw; - bool is_recursive; - bool is_broken; - bool is_linker_init; + atomic_uint32_t flags; u32 next; // in MetaMap DDMutex dd; SyncClock read_clock; // Used for rw mutexes only. @@ -61,6 +81,26 @@ struct SyncVar { *uid = id >> 48; return (uptr)GetLsb(id, 48); } + + bool IsFlagSet(u32 f) const { + return atomic_load_relaxed(&flags) & f; + } + + void SetFlags(u32 f) { + atomic_store_relaxed(&flags, atomic_load_relaxed(&flags) | f); + } + + void UpdateFlags(u32 flagz) { + // Filter out operation flags. + if (!(flagz & MutexCreationFlagMask)) + return; + u32 current = atomic_load_relaxed(&flags); + if (current & MutexCreationFlagMask) + return; + // Note: this can be called from MutexPostReadLock which holds only read + // lock on the SyncVar. + atomic_store_relaxed(&flags, current | (flagz & MutexCreationFlagMask)); + } }; /* MetaMap allows to map arbitrary user pointers onto various descriptors. diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_checks.inc b/contrib/compiler-rt/lib/ubsan/ubsan_checks.inc index 6e08641..0a87e6e 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_checks.inc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_checks.inc @@ -19,6 +19,7 @@ UBSAN_CHECK(GenericUB, "undefined-behavior", "undefined") UBSAN_CHECK(NullPointerUse, "null-pointer-use", "null") +UBSAN_CHECK(PointerOverflow, "pointer-overflow", "pointer-overflow") UBSAN_CHECK(MisalignedPointerUse, "misaligned-pointer-use", "alignment") UBSAN_CHECK(InsufficientObjectSize, "insufficient-object-size", "object-size") UBSAN_CHECK(SignedIntegerOverflow, "signed-integer-overflow", diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_diag.cc b/contrib/compiler-rt/lib/ubsan/ubsan_diag.cc index c531c5f..742802b 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_diag.cc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_diag.cc @@ -31,15 +31,16 @@ static void MaybePrintStackTrace(uptr pc, uptr bp) { // will definitely be called when we print the first diagnostics message. if (!flags()->print_stacktrace) return; - // We can only use slow unwind, as we don't have any information about stack - // top/bottom. - // FIXME: It's better to respect "fast_unwind_on_fatal" runtime flag and - // fetch stack top/bottom information if we have it (e.g. if we're running - // under ASan). - if (StackTrace::WillUseFastUnwind(false)) - return; + + uptr top = 0; + uptr bottom = 0; + bool request_fast_unwind = common_flags()->fast_unwind_on_fatal; + if (request_fast_unwind) + __sanitizer::GetThreadStackTopAndBottom(false, &top, &bottom); + BufferedStackTrace stack; - stack.Unwind(kStackTraceMax, pc, bp, 0, 0, 0, false); + stack.Unwind(kStackTraceMax, pc, bp, nullptr, top, bottom, + request_fast_unwind); stack.Print(); } @@ -79,16 +80,16 @@ static void MaybeReportErrorSummary(Location Loc, ErrorType Type) { AI.line = SLoc.getLine(); AI.column = SLoc.getColumn(); AI.function = internal_strdup(""); // Avoid printing ?? as function name. - ReportErrorSummary(ErrorKind, AI); + ReportErrorSummary(ErrorKind, AI, GetSanititizerToolName()); AI.Clear(); return; } } else if (Loc.isSymbolizedStack()) { const AddressInfo &AI = Loc.getSymbolizedStack()->info; - ReportErrorSummary(ErrorKind, AI); + ReportErrorSummary(ErrorKind, AI, GetSanititizerToolName()); return; } - ReportErrorSummary(ErrorKind); + ReportErrorSummary(ErrorKind, GetSanititizerToolName()); } namespace { diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_diag_standalone.cc b/contrib/compiler-rt/lib/ubsan/ubsan_diag_standalone.cc new file mode 100644 index 0000000..df8ed5f --- /dev/null +++ b/contrib/compiler-rt/lib/ubsan/ubsan_diag_standalone.cc @@ -0,0 +1,37 @@ +//===-- ubsan_diag_standalone.cc ------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// Diagnostic reporting for the standalone UBSan runtime. +// +//===----------------------------------------------------------------------===// + +#include "ubsan_platform.h" +#if CAN_SANITIZE_UB +#include "ubsan_diag.h" + +using namespace __ubsan; + +extern "C" { +SANITIZER_INTERFACE_ATTRIBUTE +void __sanitizer_print_stack_trace() { + uptr top = 0; + uptr bottom = 0; + bool request_fast_unwind = common_flags()->fast_unwind_on_fatal; + if (request_fast_unwind) + __sanitizer::GetThreadStackTopAndBottom(false, &top, &bottom); + + GET_REPORT_OPTIONS(false); + BufferedStackTrace stack; + stack.Unwind(kStackTraceMax, Opts.pc, Opts.bp, nullptr, top, bottom, + request_fast_unwind); + stack.Print(); +} +} // extern "C" + +#endif // CAN_SANITIZE_UB diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_flags.cc b/contrib/compiler-rt/lib/ubsan/ubsan_flags.cc index e77ba55..8e1f408 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_flags.cc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_flags.cc @@ -45,6 +45,7 @@ void InitializeFlags() { CommonFlags cf; cf.CopyFrom(*common_flags()); cf.print_summary = false; + cf.external_symbolizer_path = GetEnv("UBSAN_SYMBOLIZER_PATH"); OverrideCommonFlags(cf); } @@ -67,22 +68,8 @@ void InitializeFlags() { } // namespace __ubsan -extern "C" { - -#if !SANITIZER_SUPPORTS_WEAK_HOOKS -SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -const char *__ubsan_default_options() { return ""; } -#endif - -#if SANITIZER_WINDOWS -const char *__ubsan_default_default_options() { return ""; } -# ifdef _WIN64 -# pragma comment(linker, "/alternatename:__ubsan_default_options=__ubsan_default_default_options") -# else -# pragma comment(linker, "/alternatename:___ubsan_default_options=___ubsan_default_default_options") -# endif -#endif - -} // extern "C" +SANITIZER_INTERFACE_WEAK_DEF(const char *, __ubsan_default_options, void) { + return ""; +} #endif // CAN_SANITIZE_UB diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_handlers.cc b/contrib/compiler-rt/lib/ubsan/ubsan_handlers.cc index 6ffffae..75a4490 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_handlers.cc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_handlers.cc @@ -38,7 +38,7 @@ bool ignoreReport(SourceLocation SLoc, ReportOptions Opts, ErrorType ET) { const char *TypeCheckKinds[] = { "load of", "store to", "reference binding to", "member access within", "member call on", "constructor call on", "downcast of", "downcast of", - "upcast of", "cast to virtual base of"}; + "upcast of", "cast to virtual base of", "_Nonnull binding to"}; } static void handleTypeMismatchImpl(TypeMismatchData *Data, ValueHandle Pointer, @@ -390,7 +390,7 @@ static void handleFloatCastOverflow(void *DataPtr, ValueHandle From, ScopedReport R(Opts, Loc, ET); Diag(Loc, DL_Error, - "value %0 is outside the range of representable values of type %2") + "%0 is outside the range of representable values of type %2") << Value(*FromType, From) << *FromType << *ToType; } @@ -410,7 +410,8 @@ static void handleLoadInvalidValue(InvalidValueData *Data, ValueHandle Val, SourceLocation Loc = Data->Loc.acquire(); // This check could be more precise if we used different handlers for // -fsanitize=bool and -fsanitize=enum. - bool IsBool = (0 == internal_strcmp(Data->Type.getTypeName(), "'bool'")); + bool IsBool = (0 == internal_strcmp(Data->Type.getTypeName(), "'bool'")) || + (0 == internal_strncmp(Data->Type.getTypeName(), "'BOOL'", 6)); ErrorType ET = IsBool ? ErrorType::InvalidBoolLoad : ErrorType::InvalidEnumLoad; @@ -472,8 +473,12 @@ void __ubsan::__ubsan_handle_function_type_mismatch_abort( Die(); } -static void handleNonNullReturn(NonNullReturnData *Data, ReportOptions Opts) { - SourceLocation Loc = Data->Loc.acquire(); +static void handleNonNullReturn(NonNullReturnData *Data, SourceLocation *LocPtr, + ReportOptions Opts, bool IsAttr) { + if (!LocPtr) + UNREACHABLE("source location pointer is null!"); + + SourceLocation Loc = LocPtr->acquire(); ErrorType ET = ErrorType::InvalidNullReturn; if (ignoreReport(Loc, Opts, ET)) @@ -484,21 +489,39 @@ static void handleNonNullReturn(NonNullReturnData *Data, ReportOptions Opts) { Diag(Loc, DL_Error, "null pointer returned from function declared to never " "return null"); if (!Data->AttrLoc.isInvalid()) - Diag(Data->AttrLoc, DL_Note, "returns_nonnull attribute specified here"); + Diag(Data->AttrLoc, DL_Note, "%0 specified here") + << (IsAttr ? "returns_nonnull attribute" + : "_Nonnull return type annotation"); +} + +void __ubsan::__ubsan_handle_nonnull_return_v1(NonNullReturnData *Data, + SourceLocation *LocPtr) { + GET_REPORT_OPTIONS(false); + handleNonNullReturn(Data, LocPtr, Opts, true); } -void __ubsan::__ubsan_handle_nonnull_return(NonNullReturnData *Data) { +void __ubsan::__ubsan_handle_nonnull_return_v1_abort(NonNullReturnData *Data, + SourceLocation *LocPtr) { + GET_REPORT_OPTIONS(true); + handleNonNullReturn(Data, LocPtr, Opts, true); + Die(); +} + +void __ubsan::__ubsan_handle_nullability_return_v1(NonNullReturnData *Data, + SourceLocation *LocPtr) { GET_REPORT_OPTIONS(false); - handleNonNullReturn(Data, Opts); + handleNonNullReturn(Data, LocPtr, Opts, false); } -void __ubsan::__ubsan_handle_nonnull_return_abort(NonNullReturnData *Data) { +void __ubsan::__ubsan_handle_nullability_return_v1_abort( + NonNullReturnData *Data, SourceLocation *LocPtr) { GET_REPORT_OPTIONS(true); - handleNonNullReturn(Data, Opts); + handleNonNullReturn(Data, LocPtr, Opts, false); Die(); } -static void handleNonNullArg(NonNullArgData *Data, ReportOptions Opts) { +static void handleNonNullArg(NonNullArgData *Data, ReportOptions Opts, + bool IsAttr) { SourceLocation Loc = Data->Loc.acquire(); ErrorType ET = ErrorType::InvalidNullArgument; @@ -507,20 +530,76 @@ static void handleNonNullArg(NonNullArgData *Data, ReportOptions Opts) { ScopedReport R(Opts, Loc, ET); - Diag(Loc, DL_Error, "null pointer passed as argument %0, which is declared to " - "never be null") << Data->ArgIndex; + Diag(Loc, DL_Error, + "null pointer passed as argument %0, which is declared to " + "never be null") + << Data->ArgIndex; if (!Data->AttrLoc.isInvalid()) - Diag(Data->AttrLoc, DL_Note, "nonnull attribute specified here"); + Diag(Data->AttrLoc, DL_Note, "%0 specified here") + << (IsAttr ? "nonnull attribute" : "_Nonnull type annotation"); } void __ubsan::__ubsan_handle_nonnull_arg(NonNullArgData *Data) { GET_REPORT_OPTIONS(false); - handleNonNullArg(Data, Opts); + handleNonNullArg(Data, Opts, true); } void __ubsan::__ubsan_handle_nonnull_arg_abort(NonNullArgData *Data) { GET_REPORT_OPTIONS(true); - handleNonNullArg(Data, Opts); + handleNonNullArg(Data, Opts, true); + Die(); +} + +void __ubsan::__ubsan_handle_nullability_arg(NonNullArgData *Data) { + GET_REPORT_OPTIONS(false); + handleNonNullArg(Data, Opts, false); +} + +void __ubsan::__ubsan_handle_nullability_arg_abort(NonNullArgData *Data) { + GET_REPORT_OPTIONS(true); + handleNonNullArg(Data, Opts, false); + Die(); +} + +static void handlePointerOverflowImpl(PointerOverflowData *Data, + ValueHandle Base, + ValueHandle Result, + ReportOptions Opts) { + SourceLocation Loc = Data->Loc.acquire(); + ErrorType ET = ErrorType::PointerOverflow; + + if (ignoreReport(Loc, Opts, ET)) + return; + + ScopedReport R(Opts, Loc, ET); + + if ((sptr(Base) >= 0) == (sptr(Result) >= 0)) { + if (Base > Result) + Diag(Loc, DL_Error, "addition of unsigned offset to %0 overflowed to %1") + << (void *)Base << (void *)Result; + else + Diag(Loc, DL_Error, + "subtraction of unsigned offset from %0 overflowed to %1") + << (void *)Base << (void *)Result; + } else { + Diag(Loc, DL_Error, + "pointer index expression with base %0 overflowed to %1") + << (void *)Base << (void *)Result; + } +} + +void __ubsan::__ubsan_handle_pointer_overflow(PointerOverflowData *Data, + ValueHandle Base, + ValueHandle Result) { + GET_REPORT_OPTIONS(false); + handlePointerOverflowImpl(Data, Base, Result, Opts); +} + +void __ubsan::__ubsan_handle_pointer_overflow_abort(PointerOverflowData *Data, + ValueHandle Base, + ValueHandle Result) { + GET_REPORT_OPTIONS(true); + handlePointerOverflowImpl(Data, Base, Result, Opts); Die(); } diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_handlers.h b/contrib/compiler-rt/lib/ubsan/ubsan_handlers.h index 350eb91..796321b 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_handlers.h +++ b/contrib/compiler-rt/lib/ubsan/ubsan_handlers.h @@ -132,12 +132,13 @@ RECOVERABLE(function_type_mismatch, ValueHandle Val) struct NonNullReturnData { - SourceLocation Loc; SourceLocation AttrLoc; }; -/// \brief Handle returning null from function with returns_nonnull attribute. -RECOVERABLE(nonnull_return, NonNullReturnData *Data) +/// \brief Handle returning null from function with the returns_nonnull +/// attribute, or a return type annotated with _Nonnull. +RECOVERABLE(nonnull_return_v1, NonNullReturnData *Data, SourceLocation *Loc) +RECOVERABLE(nullability_return_v1, NonNullReturnData *Data, SourceLocation *Loc) struct NonNullArgData { SourceLocation Loc; @@ -145,8 +146,17 @@ struct NonNullArgData { int ArgIndex; }; -/// \brief Handle passing null pointer to function with nonnull attribute. +/// \brief Handle passing null pointer to a function parameter with the nonnull +/// attribute, or a _Nonnull type annotation. RECOVERABLE(nonnull_arg, NonNullArgData *Data) +RECOVERABLE(nullability_arg, NonNullArgData *Data) + +struct PointerOverflowData { + SourceLocation Loc; +}; + +RECOVERABLE(pointer_overflow, PointerOverflowData *Data, ValueHandle Base, + ValueHandle Result) /// \brief Known CFI check kinds. /// Keep in sync with the enum of the same name in CodeGenFunction.h diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_init.cc b/contrib/compiler-rt/lib/ubsan/ubsan_init.cc index b4f42c4..307bca3 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_init.cc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_init.cc @@ -23,6 +23,10 @@ using namespace __ubsan; +const char *__ubsan::GetSanititizerToolName() { + return "UndefinedBehaviorSanitizer"; +} + static enum { UBSAN_MODE_UNKNOWN = 0, UBSAN_MODE_STANDALONE, @@ -35,7 +39,7 @@ static void CommonInit() { } static void CommonStandaloneInit() { - SanitizerToolName = "UndefinedBehaviorSanitizer"; + SanitizerToolName = GetSanititizerToolName(); InitializeFlags(); CacheBinaryName(); __sanitizer_set_report_path(common_flags()->log_path); diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_init.h b/contrib/compiler-rt/lib/ubsan/ubsan_init.h index 103ae24..f12fc2c 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_init.h +++ b/contrib/compiler-rt/lib/ubsan/ubsan_init.h @@ -15,6 +15,9 @@ namespace __ubsan { +// Get the full tool name for UBSan. +const char *GetSanititizerToolName(); + // Initialize UBSan as a standalone tool. Typically should be called early // during initialization. void InitAsStandalone(); diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_interface.inc b/contrib/compiler-rt/lib/ubsan/ubsan_interface.inc new file mode 100644 index 0000000..a69ca57 --- /dev/null +++ b/contrib/compiler-rt/lib/ubsan/ubsan_interface.inc @@ -0,0 +1,49 @@ +//===-- ubsan_interface.inc -----------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// Ubsan interface list. +//===----------------------------------------------------------------------===// +INTERFACE_FUNCTION(__ubsan_handle_add_overflow) +INTERFACE_FUNCTION(__ubsan_handle_add_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_builtin_unreachable) +INTERFACE_FUNCTION(__ubsan_handle_cfi_check_fail) +INTERFACE_FUNCTION(__ubsan_handle_cfi_check_fail_abort) +INTERFACE_FUNCTION(__ubsan_handle_divrem_overflow) +INTERFACE_FUNCTION(__ubsan_handle_divrem_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_float_cast_overflow) +INTERFACE_FUNCTION(__ubsan_handle_float_cast_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_function_type_mismatch) +INTERFACE_FUNCTION(__ubsan_handle_function_type_mismatch_abort) +INTERFACE_FUNCTION(__ubsan_handle_load_invalid_value) +INTERFACE_FUNCTION(__ubsan_handle_load_invalid_value_abort) +INTERFACE_FUNCTION(__ubsan_handle_missing_return) +INTERFACE_FUNCTION(__ubsan_handle_mul_overflow) +INTERFACE_FUNCTION(__ubsan_handle_mul_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_negate_overflow) +INTERFACE_FUNCTION(__ubsan_handle_negate_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_nonnull_arg) +INTERFACE_FUNCTION(__ubsan_handle_nonnull_arg_abort) +INTERFACE_FUNCTION(__ubsan_handle_nonnull_return_v1) +INTERFACE_FUNCTION(__ubsan_handle_nonnull_return_v1_abort) +INTERFACE_FUNCTION(__ubsan_handle_nullability_arg) +INTERFACE_FUNCTION(__ubsan_handle_nullability_arg_abort) +INTERFACE_FUNCTION(__ubsan_handle_nullability_return_v1) +INTERFACE_FUNCTION(__ubsan_handle_nullability_return_v1_abort) +INTERFACE_FUNCTION(__ubsan_handle_out_of_bounds) +INTERFACE_FUNCTION(__ubsan_handle_out_of_bounds_abort) +INTERFACE_FUNCTION(__ubsan_handle_pointer_overflow) +INTERFACE_FUNCTION(__ubsan_handle_pointer_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_shift_out_of_bounds) +INTERFACE_FUNCTION(__ubsan_handle_shift_out_of_bounds_abort) +INTERFACE_FUNCTION(__ubsan_handle_sub_overflow) +INTERFACE_FUNCTION(__ubsan_handle_sub_overflow_abort) +INTERFACE_FUNCTION(__ubsan_handle_type_mismatch_v1) +INTERFACE_FUNCTION(__ubsan_handle_type_mismatch_v1_abort) +INTERFACE_FUNCTION(__ubsan_handle_vla_bound_not_positive) +INTERFACE_FUNCTION(__ubsan_handle_vla_bound_not_positive_abort) +INTERFACE_WEAK_FUNCTION(__ubsan_default_options) diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc b/contrib/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc index 5ae5ae0..dcce0dd 100644 --- a/contrib/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc +++ b/contrib/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc @@ -197,9 +197,9 @@ struct VtablePrefix { }; VtablePrefix *getVtablePrefix(void *Vtable) { VtablePrefix *Vptr = reinterpret_cast<VtablePrefix*>(Vtable); - if (!Vptr) - return nullptr; VtablePrefix *Prefix = Vptr - 1; + if (!IsAccessibleMemoryRange((uptr)Prefix, sizeof(VtablePrefix))) + return nullptr; if (!Prefix->TypeInfo) // This can't possibly be a valid vtable. return nullptr; diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_win_dll_thunk.cc b/contrib/compiler-rt/lib/ubsan/ubsan_win_dll_thunk.cc new file mode 100644 index 0000000..a1d0dbd --- /dev/null +++ b/contrib/compiler-rt/lib/ubsan/ubsan_win_dll_thunk.cc @@ -0,0 +1,21 @@ +//===-- ubsan_win_dll_thunk.cc --------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines a family of thunks that should be statically linked into +// the DLLs that have instrumentation in order to delegate the calls to the +// shared runtime that lives in the main binary. +// See https://github.com/google/sanitizers/issues/209 for the details. +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DLL_THUNK +#include "sanitizer_common/sanitizer_win_dll_thunk.h" +// Ubsan interface functions. +#define INTERFACE_FUNCTION(Name) INTERCEPT_SANITIZER_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "ubsan_interface.inc" +#endif // SANITIZER_DLL_THUNK diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_win_dynamic_runtime_thunk.cc b/contrib/compiler-rt/lib/ubsan/ubsan_win_dynamic_runtime_thunk.cc new file mode 100644 index 0000000..c9b74a4 --- /dev/null +++ b/contrib/compiler-rt/lib/ubsan/ubsan_win_dynamic_runtime_thunk.cc @@ -0,0 +1,21 @@ +//===-- ubsan_win_dynamic_runtime_thunk.cc --------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file defines things that need to be present in the application modules +// to interact with Ubsan, when it is included in a dll. +// +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC_RUNTIME_THUNK +#define SANITIZER_IMPORT_INTERFACE 1 +#include "sanitizer_common/sanitizer_win_defs.h" +// Define weak alias for all weak functions imported from ubsan. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) WIN_WEAK_IMPORT_DEF(Name) +#include "ubsan_interface.inc" +#endif // SANITIZER_DYNAMIC_RUNTIME_THUNK diff --git a/contrib/compiler-rt/lib/ubsan/ubsan_win_weak_interception.cc b/contrib/compiler-rt/lib/ubsan/ubsan_win_weak_interception.cc new file mode 100644 index 0000000..353719e --- /dev/null +++ b/contrib/compiler-rt/lib/ubsan/ubsan_win_weak_interception.cc @@ -0,0 +1,23 @@ +//===-- ubsan_win_weak_interception.cc ------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// This module should be included in Ubsan when it is implemented as a shared +// library on Windows (dll), in order to delegate the calls of weak functions to +// the implementation in the main executable when a strong definition is +// provided. +//===----------------------------------------------------------------------===// +#ifdef SANITIZER_DYNAMIC +#include "sanitizer_common/sanitizer_win_weak_interception.h" +#include "ubsan_flags.h" +// Check if strong definitions for weak functions are present in the main +// executable. If that is the case, override dll functions to point to strong +// implementations. +#define INTERFACE_FUNCTION(Name) +#define INTERFACE_WEAK_FUNCTION(Name) INTERCEPT_SANITIZER_WEAK_FUNCTION(Name) +#include "ubsan_interface.inc" +#endif // SANITIZER_DYNAMIC diff --git a/contrib/compiler-rt/lib/xray/xray_AArch64.cc b/contrib/compiler-rt/lib/xray/xray_AArch64.cc index 0c1df22..f26e77d 100644 --- a/contrib/compiler-rt/lib/xray/xray_AArch64.cc +++ b/contrib/compiler-rt/lib/xray/xray_AArch64.cc @@ -14,29 +14,14 @@ //===----------------------------------------------------------------------===// #include "sanitizer_common/sanitizer_common.h" #include "xray_defs.h" -#include "xray_emulate_tsc.h" #include "xray_interface_internal.h" #include <atomic> #include <cassert> - -extern "C" void __clear_cache(void* start, void* end); +extern "C" void __clear_cache(void *start, void *end); namespace __xray { -uint64_t cycleFrequency() XRAY_NEVER_INSTRUMENT { - // There is no instruction like RDTSCP in user mode on ARM. ARM's CP15 does - // not have a constant frequency like TSC on x86[_64]; it may go faster or - // slower depending on CPU's turbo or power saving modes. Furthermore, to - // read from CP15 on ARM a kernel modification or a driver is needed. - // We can not require this from users of compiler-rt. - // So on ARM we use clock_gettime(2) which gives the result in nanoseconds. - // To get the measurements per second, we scale this by the number of - // nanoseconds per second, pretending that the TSC frequency is 1GHz and - // one TSC tick is 1 nanosecond. - return NanosecondsPerSecond; -} - // The machine codes for some instructions used in runtime patching. enum class PatchOpcodes : uint32_t { PO_StpX0X30SP_m16e = 0xA9BF7BE0, // STP X0, X30, [SP, #-16]! @@ -100,14 +85,15 @@ inline static bool patchSled(const bool Enable, const uint32_t FuncId, reinterpret_cast<std::atomic<uint32_t> *>(FirstAddress), uint32_t(PatchOpcodes::PO_B32), std::memory_order_release); } - __clear_cache(reinterpret_cast<char*>(FirstAddress), - reinterpret_cast<char*>(CurAddress)); + __clear_cache(reinterpret_cast<char *>(FirstAddress), + reinterpret_cast<char *>(CurAddress)); return true; } bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, - const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { - return patchSled(Enable, FuncId, Sled, __xray_FunctionEntry); + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, Trampoline); } bool patchFunctionExit(const bool Enable, const uint32_t FuncId, @@ -117,9 +103,20 @@ bool patchFunctionExit(const bool Enable, const uint32_t FuncId, bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { - // FIXME: In the future we'd need to distinguish between non-tail exits and - // tail exits for better information preservation. - return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); + return patchSled(Enable, FuncId, Sled, __xray_FunctionTailExit); } +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) + XRAY_NEVER_INSTRUMENT { // FIXME: Implement in aarch64? + return false; +} + +// FIXME: Maybe implement this better? +bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; } + } // namespace __xray + +extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT { + // FIXME: this will have to be implemented in the trampoline assembly file +} diff --git a/contrib/compiler-rt/lib/xray/xray_always_instrument.txt b/contrib/compiler-rt/lib/xray/xray_always_instrument.txt new file mode 100644 index 0000000..151ed70 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_always_instrument.txt @@ -0,0 +1,6 @@ +# List of function matchers common to C/C++ applications that make sense to +# always instrument. You can use this as an argument to +# -fxray-always-instrument=<path> along with your project-specific lists. + +# Always instrument the main function. +fun:main diff --git a/contrib/compiler-rt/lib/xray/xray_arm.cc b/contrib/compiler-rt/lib/xray/xray_arm.cc index f5e2cd2..da4efcd 100644 --- a/contrib/compiler-rt/lib/xray/xray_arm.cc +++ b/contrib/compiler-rt/lib/xray/xray_arm.cc @@ -14,28 +14,14 @@ //===----------------------------------------------------------------------===// #include "sanitizer_common/sanitizer_common.h" #include "xray_defs.h" -#include "xray_emulate_tsc.h" #include "xray_interface_internal.h" #include <atomic> #include <cassert> -extern "C" void __clear_cache(void* start, void* end); +extern "C" void __clear_cache(void *start, void *end); namespace __xray { -uint64_t cycleFrequency() XRAY_NEVER_INSTRUMENT { - // There is no instruction like RDTSCP in user mode on ARM. ARM's CP15 does - // not have a constant frequency like TSC on x86[_64]; it may go faster or - // slower depending on CPU's turbo or power saving modes. Furthermore, to - // read from CP15 on ARM a kernel modification or a driver is needed. - // We can not require this from users of compiler-rt. - // So on ARM we use clock_gettime(2) which gives the result in nanoseconds. - // To get the measurements per second, we scale this by the number of - // nanoseconds per second, pretending that the TSC frequency is 1GHz and - // one TSC tick is 1 nanosecond. - return NanosecondsPerSecond; -} - // The machine codes for some instructions used in runtime patching. enum class PatchOpcodes : uint32_t { PO_PushR0Lr = 0xE92D4001, // PUSH {r0, lr} @@ -74,7 +60,7 @@ write32bitLoadReg(uint8_t regNo, uint32_t *Address, // MOVW r0, #<lower 16 bits of the |Value|> // MOVT r0, #<higher 16 bits of the |Value|> inline static uint32_t * -Write32bitLoadR0(uint32_t *Address, +write32bitLoadR0(uint32_t *Address, const uint32_t Value) XRAY_NEVER_INSTRUMENT { return write32bitLoadReg(0, Address, Value); } @@ -83,7 +69,7 @@ Write32bitLoadR0(uint32_t *Address, // MOVW ip, #<lower 16 bits of the |Value|> // MOVT ip, #<higher 16 bits of the |Value|> inline static uint32_t * -Write32bitLoadIP(uint32_t *Address, +write32bitLoadIP(uint32_t *Address, const uint32_t Value) XRAY_NEVER_INSTRUMENT { return write32bitLoadReg(12, Address, Value); } @@ -121,9 +107,9 @@ inline static bool patchSled(const bool Enable, const uint32_t FuncId, uint32_t *CurAddress = FirstAddress + 1; if (Enable) { CurAddress = - Write32bitLoadR0(CurAddress, reinterpret_cast<uint32_t>(FuncId)); + write32bitLoadR0(CurAddress, reinterpret_cast<uint32_t>(FuncId)); CurAddress = - Write32bitLoadIP(CurAddress, reinterpret_cast<uint32_t>(TracingHook)); + write32bitLoadIP(CurAddress, reinterpret_cast<uint32_t>(TracingHook)); *CurAddress = uint32_t(PatchOpcodes::PO_BlxIp); CurAddress++; *CurAddress = uint32_t(PatchOpcodes::PO_PopR0Lr); @@ -136,14 +122,15 @@ inline static bool patchSled(const bool Enable, const uint32_t FuncId, reinterpret_cast<std::atomic<uint32_t> *>(FirstAddress), uint32_t(PatchOpcodes::PO_B20), std::memory_order_release); } - __clear_cache(reinterpret_cast<char*>(FirstAddress), - reinterpret_cast<char*>(CurAddress)); + __clear_cache(reinterpret_cast<char *>(FirstAddress), + reinterpret_cast<char *>(CurAddress)); return true; } bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, - const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { - return patchSled(Enable, FuncId, Sled, __xray_FunctionEntry); + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, Trampoline); } bool patchFunctionExit(const bool Enable, const uint32_t FuncId, @@ -153,9 +140,20 @@ bool patchFunctionExit(const bool Enable, const uint32_t FuncId, bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { - // FIXME: In the future we'd need to distinguish between non-tail exits and - // tail exits for better information preservation. - return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); + return patchSled(Enable, FuncId, Sled, __xray_FunctionTailExit); +} + +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) + XRAY_NEVER_INSTRUMENT { // FIXME: Implement in arm? + return false; } +// FIXME: Maybe implement this better? +bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; } + } // namespace __xray + +extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT { + // FIXME: this will have to be implemented in the trampoline assembly file +} diff --git a/contrib/compiler-rt/lib/xray/xray_buffer_queue.cc b/contrib/compiler-rt/lib/xray/xray_buffer_queue.cc index 7e5462f..7ba755a 100644 --- a/contrib/compiler-rt/lib/xray/xray_buffer_queue.cc +++ b/contrib/compiler-rt/lib/xray/xray_buffer_queue.cc @@ -13,53 +13,69 @@ // //===----------------------------------------------------------------------===// #include "xray_buffer_queue.h" -#include <cassert> +#include "sanitizer_common/sanitizer_common.h" +#include "sanitizer_common/sanitizer_libc.h" + #include <cstdlib> +#include <tuple> using namespace __xray; +using namespace __sanitizer; -BufferQueue::BufferQueue(std::size_t B, std::size_t N) - : BufferSize(B), Buffers(N), Mutex(), OwnedBuffers(), Finalizing(false) { - for (auto &Buf : Buffers) { +BufferQueue::BufferQueue(std::size_t B, std::size_t N, bool &Success) + : BufferSize(B), Buffers(N), Mutex(), OwnedBuffers(), Finalizing{0} { + for (auto &T : Buffers) { void *Tmp = malloc(BufferSize); + if (Tmp == nullptr) { + Success = false; + return; + } + + auto &Buf = std::get<0>(T); Buf.Buffer = Tmp; Buf.Size = B; - if (Tmp != 0) - OwnedBuffers.insert(Tmp); + OwnedBuffers.emplace(Tmp); } + Success = true; } -std::error_code BufferQueue::getBuffer(Buffer &Buf) { - if (Finalizing.load(std::memory_order_acquire)) - return std::make_error_code(std::errc::state_not_recoverable); - std::lock_guard<std::mutex> Guard(Mutex); +BufferQueue::ErrorCode BufferQueue::getBuffer(Buffer &Buf) { + if (__sanitizer::atomic_load(&Finalizing, __sanitizer::memory_order_acquire)) + return ErrorCode::QueueFinalizing; + __sanitizer::BlockingMutexLock Guard(&Mutex); if (Buffers.empty()) - return std::make_error_code(std::errc::not_enough_memory); - Buf = Buffers.front(); + return ErrorCode::NotEnoughMemory; + auto &T = Buffers.front(); + auto &B = std::get<0>(T); + Buf = B; + B.Buffer = nullptr; + B.Size = 0; Buffers.pop_front(); - return {}; + return ErrorCode::Ok; } -std::error_code BufferQueue::releaseBuffer(Buffer &Buf) { +BufferQueue::ErrorCode BufferQueue::releaseBuffer(Buffer &Buf) { if (OwnedBuffers.count(Buf.Buffer) == 0) - return std::make_error_code(std::errc::argument_out_of_domain); - std::lock_guard<std::mutex> Guard(Mutex); - Buffers.push_back(Buf); + return ErrorCode::UnrecognizedBuffer; + __sanitizer::BlockingMutexLock Guard(&Mutex); + + // Now that the buffer has been released, we mark it as "used". + Buffers.emplace(Buffers.end(), Buf, true /* used */); Buf.Buffer = nullptr; - Buf.Size = BufferSize; - return {}; + Buf.Size = 0; + return ErrorCode::Ok; } -std::error_code BufferQueue::finalize() { - if (Finalizing.exchange(true, std::memory_order_acq_rel)) - return std::make_error_code(std::errc::state_not_recoverable); - return {}; +BufferQueue::ErrorCode BufferQueue::finalize() { + if (__sanitizer::atomic_exchange(&Finalizing, 1, + __sanitizer::memory_order_acq_rel)) + return ErrorCode::QueueFinalizing; + return ErrorCode::Ok; } BufferQueue::~BufferQueue() { - for (auto &Buf : Buffers) { + for (auto &T : Buffers) { + auto &Buf = std::get<0>(T); free(Buf.Buffer); - Buf.Buffer = nullptr; - Buf.Size = 0; } } diff --git a/contrib/compiler-rt/lib/xray/xray_buffer_queue.h b/contrib/compiler-rt/lib/xray/xray_buffer_queue.h index bf0b7af..e051695 100644 --- a/contrib/compiler-rt/lib/xray/xray_buffer_queue.h +++ b/contrib/compiler-rt/lib/xray/xray_buffer_queue.h @@ -15,12 +15,11 @@ #ifndef XRAY_BUFFER_QUEUE_H #define XRAY_BUFFER_QUEUE_H -#include <atomic> -#include <cstdint> +#include "sanitizer_common/sanitizer_atomic.h" +#include "sanitizer_common/sanitizer_mutex.h" #include <deque> -#include <mutex> -#include <system_error> #include <unordered_set> +#include <utility> namespace __xray { @@ -33,19 +32,47 @@ class BufferQueue { public: struct Buffer { void *Buffer = nullptr; - std::size_t Size = 0; + size_t Size = 0; }; private: - std::size_t BufferSize; - std::deque<Buffer> Buffers; - std::mutex Mutex; + size_t BufferSize; + + // We use a bool to indicate whether the Buffer has been used in this + // freelist implementation. + std::deque<std::tuple<Buffer, bool>> Buffers; + __sanitizer::BlockingMutex Mutex; std::unordered_set<void *> OwnedBuffers; - std::atomic<bool> Finalizing; + __sanitizer::atomic_uint8_t Finalizing; public: - /// Initialise a queue of size |N| with buffers of size |B|. - BufferQueue(std::size_t B, std::size_t N); + enum class ErrorCode : unsigned { + Ok, + NotEnoughMemory, + QueueFinalizing, + UnrecognizedBuffer, + AlreadyFinalized, + }; + + static const char *getErrorString(ErrorCode E) { + switch (E) { + case ErrorCode::Ok: + return "(none)"; + case ErrorCode::NotEnoughMemory: + return "no available buffers in the queue"; + case ErrorCode::QueueFinalizing: + return "queue already finalizing"; + case ErrorCode::UnrecognizedBuffer: + return "buffer being returned not owned by buffer queue"; + case ErrorCode::AlreadyFinalized: + return "queue already finalized"; + } + return "unknown error"; + } + + /// Initialise a queue of size |N| with buffers of size |B|. We report success + /// through |Success|. + BufferQueue(size_t B, size_t N, bool &Success); /// Updates |Buf| to contain the pointer to an appropriate buffer. Returns an /// error in case there are no available buffers to return when we will run @@ -58,24 +85,41 @@ public: /// - std::errc::not_enough_memory on exceeding MaxSize. /// - no error when we find a Buffer. /// - std::errc::state_not_recoverable on finalising BufferQueue. - std::error_code getBuffer(Buffer &Buf); + ErrorCode getBuffer(Buffer &Buf); /// Updates |Buf| to point to nullptr, with size 0. /// /// Returns: /// - ... - std::error_code releaseBuffer(Buffer &Buf); - - bool finalizing() const { return Finalizing.load(std::memory_order_acquire); } - - // Sets the state of the BufferQueue to finalizing, which ensures that: - // - // - All subsequent attempts to retrieve a Buffer will fail. - // - All releaseBuffer operations will not fail. - // - // After a call to finalize succeeds, all subsequent calls to finalize will - // fail with std::errc::state_not_recoverable. - std::error_code finalize(); + ErrorCode releaseBuffer(Buffer &Buf); + + bool finalizing() const { + return __sanitizer::atomic_load(&Finalizing, + __sanitizer::memory_order_acquire); + } + + /// Returns the configured size of the buffers in the buffer queue. + size_t ConfiguredBufferSize() const { return BufferSize; } + + /// Sets the state of the BufferQueue to finalizing, which ensures that: + /// + /// - All subsequent attempts to retrieve a Buffer will fail. + /// - All releaseBuffer operations will not fail. + /// + /// After a call to finalize succeeds, all subsequent calls to finalize will + /// fail with std::errc::state_not_recoverable. + ErrorCode finalize(); + + /// Applies the provided function F to each Buffer in the queue, only if the + /// Buffer is marked 'used' (i.e. has been the result of getBuffer(...) and a + /// releaseBuffer(...) operation. + template <class F> void apply(F Fn) { + __sanitizer::BlockingMutexLock G(&Mutex); + for (const auto &T : Buffers) { + if (std::get<1>(T)) + Fn(std::get<0>(T)); + } + } // Cleans up allocated buffers. ~BufferQueue(); diff --git a/contrib/compiler-rt/lib/xray/xray_emulate_tsc.h b/contrib/compiler-rt/lib/xray/xray_emulate_tsc.h deleted file mode 100644 index a3e8b1c..0000000 --- a/contrib/compiler-rt/lib/xray/xray_emulate_tsc.h +++ /dev/null @@ -1,40 +0,0 @@ -//===-- xray_emulate_tsc.h --------------------------------------*- C++ -*-===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// This file is a part of XRay, a dynamic runtime instrumentation system. -// -//===----------------------------------------------------------------------===// -#ifndef XRAY_EMULATE_TSC_H -#define XRAY_EMULATE_TSC_H - -#include "sanitizer_common/sanitizer_common.h" -#include "sanitizer_common/sanitizer_internal_defs.h" -#include "xray_defs.h" -#include <cerrno> -#include <cstdint> -#include <time.h> - -namespace __xray { - -static constexpr uint64_t NanosecondsPerSecond = 1000ULL * 1000 * 1000; - -ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT { - timespec TS; - int result = clock_gettime(CLOCK_REALTIME, &TS); - if (result != 0) { - Report("clock_gettime(2) returned %d, errno=%d.", result, int(errno)); - TS.tv_sec = 0; - TS.tv_nsec = 0; - } - CPU = 0; - return TS.tv_sec * NanosecondsPerSecond + TS.tv_nsec; -} -} - -#endif // XRAY_EMULATE_TSC_H diff --git a/contrib/compiler-rt/lib/xray/xray_fdr_log_records.h b/contrib/compiler-rt/lib/xray/xray_fdr_log_records.h new file mode 100644 index 0000000..3d6d388 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_fdr_log_records.h @@ -0,0 +1,66 @@ +//===-- xray_fdr_log_records.h -------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a function call tracing system. +// +//===----------------------------------------------------------------------===// +#ifndef XRAY_XRAY_FDR_LOG_RECORDS_H +#define XRAY_XRAY_FDR_LOG_RECORDS_H + +enum class RecordType : uint8_t { Function, Metadata }; + +// A MetadataRecord encodes the kind of record in its first byte, and have 15 +// additional bytes in the end to hold free-form data. +struct alignas(16) MetadataRecord { + // A MetadataRecord must always have a type of 1. + /* RecordType */ uint8_t Type : 1; + + // Each kind of record is represented as a 7-bit value (even though we use an + // unsigned 8-bit enum class to do so). + enum class RecordKinds : uint8_t { + NewBuffer, + EndOfBuffer, + NewCPUId, + TSCWrap, + WalltimeMarker, + CustomEventMarker, + }; + // Use 7 bits to identify this record type. + /* RecordKinds */ uint8_t RecordKind : 7; + char Data[15]; +} __attribute__((packed)); + +static_assert(sizeof(MetadataRecord) == 16, "Wrong size for MetadataRecord."); + +struct alignas(8) FunctionRecord { + // A FunctionRecord must always have a type of 0. + /* RecordType */ uint8_t Type : 1; + enum class RecordKinds { + FunctionEnter = 0x00, + FunctionExit = 0x01, + FunctionTailExit = 0x02, + }; + /* RecordKinds */ uint8_t RecordKind : 3; + + // We only use 28 bits of the function ID, so that we can use as few bytes as + // possible. This means we only support 2^28 (268,435,456) unique function ids + // in a single binary. + int FuncId : 28; + + // We use another 4 bytes to hold the delta between the previous entry's TSC. + // In case we've found that the distance is greater than the allowable 32 bits + // (either because we are running in a different CPU and the TSC might be + // different then), we should use a MetadataRecord before this FunctionRecord + // that will contain the full TSC for that CPU, and keep this to 0. + uint32_t TSCDelta; +} __attribute__((packed)); + +static_assert(sizeof(FunctionRecord) == 8, "Wrong size for FunctionRecord."); + +#endif // XRAY_XRAY_FDR_LOG_RECORDS_H diff --git a/contrib/compiler-rt/lib/xray/xray_fdr_logging.cc b/contrib/compiler-rt/lib/xray/xray_fdr_logging.cc new file mode 100644 index 0000000..a7e1382 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_fdr_logging.cc @@ -0,0 +1,300 @@ +//===-- xray_fdr_logging.cc ------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Here we implement the Flight Data Recorder mode for XRay, where we use +// compact structures to store records in memory as well as when writing out the +// data to files. +// +//===----------------------------------------------------------------------===// +#include "xray_fdr_logging.h" +#include <algorithm> +#include <bitset> +#include <cerrno> +#include <cstring> +#include <sys/syscall.h> +#include <sys/time.h> +#include <time.h> +#include <unistd.h> +#include <unordered_map> + +#include "sanitizer_common/sanitizer_atomic.h" +#include "sanitizer_common/sanitizer_common.h" +#include "xray/xray_interface.h" +#include "xray/xray_records.h" +#include "xray_buffer_queue.h" +#include "xray_defs.h" +#include "xray_fdr_logging_impl.h" +#include "xray_flags.h" +#include "xray_tsc.h" +#include "xray_utils.h" + +namespace __xray { + +// Global BufferQueue. +std::shared_ptr<BufferQueue> BQ; + +__sanitizer::atomic_sint32_t LogFlushStatus = { + XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING}; + +FDRLoggingOptions FDROptions; + +__sanitizer::SpinMutex FDROptionsMutex; + +// Must finalize before flushing. +XRayLogFlushStatus fdrLoggingFlush() XRAY_NEVER_INSTRUMENT { + if (__sanitizer::atomic_load(&LoggingStatus, + __sanitizer::memory_order_acquire) != + XRayLogInitStatus::XRAY_LOG_FINALIZED) + return XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING; + + s32 Result = XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING; + if (!__sanitizer::atomic_compare_exchange_strong( + &LogFlushStatus, &Result, XRayLogFlushStatus::XRAY_LOG_FLUSHING, + __sanitizer::memory_order_release)) + return static_cast<XRayLogFlushStatus>(Result); + + // Make a copy of the BufferQueue pointer to prevent other threads that may be + // resetting it from blowing away the queue prematurely while we're dealing + // with it. + auto LocalBQ = BQ; + + // We write out the file in the following format: + // + // 1) We write down the XRay file header with version 1, type FDR_LOG. + // 2) Then we use the 'apply' member of the BufferQueue that's live, to + // ensure that at this point in time we write down the buffers that have + // been released (and marked "used") -- we dump the full buffer for now + // (fixed-sized) and let the tools reading the buffers deal with the data + // afterwards. + // + int Fd = -1; + { + __sanitizer::SpinMutexLock Guard(&FDROptionsMutex); + Fd = FDROptions.Fd; + } + if (Fd == -1) + Fd = getLogFD(); + if (Fd == -1) { + auto Result = XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING; + __sanitizer::atomic_store(&LogFlushStatus, Result, + __sanitizer::memory_order_release); + return Result; + } + + // Test for required CPU features and cache the cycle frequency + static bool TSCSupported = probeRequiredCPUFeatures(); + static uint64_t CycleFrequency = + TSCSupported ? getTSCFrequency() : __xray::NanosecondsPerSecond; + + XRayFileHeader Header; + Header.Version = 1; + Header.Type = FileTypes::FDR_LOG; + Header.CycleFrequency = CycleFrequency; + // FIXME: Actually check whether we have 'constant_tsc' and 'nonstop_tsc' + // before setting the values in the header. + Header.ConstantTSC = 1; + Header.NonstopTSC = 1; + Header.FdrData = FdrAdditionalHeaderData{LocalBQ->ConfiguredBufferSize()}; + retryingWriteAll(Fd, reinterpret_cast<char *>(&Header), + reinterpret_cast<char *>(&Header) + sizeof(Header)); + + LocalBQ->apply([&](const BufferQueue::Buffer &B) { + uint64_t BufferSize = B.Size; + if (BufferSize > 0) { + retryingWriteAll(Fd, reinterpret_cast<char *>(B.Buffer), + reinterpret_cast<char *>(B.Buffer) + B.Size); + } + }); + __sanitizer::atomic_store(&LogFlushStatus, + XRayLogFlushStatus::XRAY_LOG_FLUSHED, + __sanitizer::memory_order_release); + return XRayLogFlushStatus::XRAY_LOG_FLUSHED; +} + +XRayLogInitStatus fdrLoggingFinalize() XRAY_NEVER_INSTRUMENT { + s32 CurrentStatus = XRayLogInitStatus::XRAY_LOG_INITIALIZED; + if (!__sanitizer::atomic_compare_exchange_strong( + &LoggingStatus, &CurrentStatus, + XRayLogInitStatus::XRAY_LOG_FINALIZING, + __sanitizer::memory_order_release)) + return static_cast<XRayLogInitStatus>(CurrentStatus); + + // Do special things to make the log finalize itself, and not allow any more + // operations to be performed until re-initialized. + BQ->finalize(); + + __sanitizer::atomic_store(&LoggingStatus, + XRayLogInitStatus::XRAY_LOG_FINALIZED, + __sanitizer::memory_order_release); + return XRayLogInitStatus::XRAY_LOG_FINALIZED; +} + +XRayLogInitStatus fdrLoggingReset() XRAY_NEVER_INSTRUMENT { + s32 CurrentStatus = XRayLogInitStatus::XRAY_LOG_FINALIZED; + if (__sanitizer::atomic_compare_exchange_strong( + &LoggingStatus, &CurrentStatus, + XRayLogInitStatus::XRAY_LOG_INITIALIZED, + __sanitizer::memory_order_release)) + return static_cast<XRayLogInitStatus>(CurrentStatus); + + // Release the in-memory buffer queue. + BQ.reset(); + + // Spin until the flushing status is flushed. + s32 CurrentFlushingStatus = XRayLogFlushStatus::XRAY_LOG_FLUSHED; + while (__sanitizer::atomic_compare_exchange_weak( + &LogFlushStatus, &CurrentFlushingStatus, + XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING, + __sanitizer::memory_order_release)) { + if (CurrentFlushingStatus == XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING) + break; + CurrentFlushingStatus = XRayLogFlushStatus::XRAY_LOG_FLUSHED; + } + + // At this point, we know that the status is flushed, and that we can assume + return XRayLogInitStatus::XRAY_LOG_UNINITIALIZED; +} + +static std::tuple<uint64_t, unsigned char> +getTimestamp() XRAY_NEVER_INSTRUMENT { + // We want to get the TSC as early as possible, so that we can check whether + // we've seen this CPU before. We also do it before we load anything else, to + // allow for forward progress with the scheduling. + unsigned char CPU; + uint64_t TSC; + + // Test once for required CPU features + static bool TSCSupported = probeRequiredCPUFeatures(); + + if (TSCSupported) { + TSC = __xray::readTSC(CPU); + } else { + // FIXME: This code needs refactoring as it appears in multiple locations + timespec TS; + int result = clock_gettime(CLOCK_REALTIME, &TS); + if (result != 0) { + Report("clock_gettime(2) return %d, errno=%d", result, int(errno)); + TS = {0, 0}; + } + CPU = 0; + TSC = TS.tv_sec * __xray::NanosecondsPerSecond + TS.tv_nsec; + } + return std::make_tuple(TSC, CPU); +} + +void fdrLoggingHandleArg0(int32_t FuncId, + XRayEntryType Entry) XRAY_NEVER_INSTRUMENT { + auto TSC_CPU = getTimestamp(); + __xray_fdr_internal::processFunctionHook(FuncId, Entry, std::get<0>(TSC_CPU), + std::get<1>(TSC_CPU), clock_gettime, + LoggingStatus, BQ); +} + +void fdrLoggingHandleCustomEvent(void *Event, + std::size_t EventSize) XRAY_NEVER_INSTRUMENT { + using namespace __xray_fdr_internal; + auto TSC_CPU = getTimestamp(); + auto &TSC = std::get<0>(TSC_CPU); + auto &CPU = std::get<1>(TSC_CPU); + thread_local bool Running = false; + RecursionGuard Guard{Running}; + if (!Guard) { + assert(Running && "RecursionGuard is buggy!"); + return; + } + if (EventSize > std::numeric_limits<int32_t>::max()) { + using Empty = struct {}; + static Empty Once = [&] { + Report("Event size too large = %zu ; > max = %d\n", EventSize, + std::numeric_limits<int32_t>::max()); + return Empty(); + }(); + (void)Once; + } + int32_t ReducedEventSize = static_cast<int32_t>(EventSize); + if (!isLogInitializedAndReady(LocalBQ, TSC, CPU, clock_gettime)) + return; + + // Here we need to prepare the log to handle: + // - The metadata record we're going to write. (16 bytes) + // - The additional data we're going to write. Currently, that's the size of + // the event we're going to dump into the log as free-form bytes. + if (!prepareBuffer(clock_gettime, MetadataRecSize + EventSize)) { + LocalBQ = nullptr; + return; + } + + // Write the custom event metadata record, which consists of the following + // information: + // - 8 bytes (64-bits) for the full TSC when the event started. + // - 4 bytes (32-bits) for the length of the data. + MetadataRecord CustomEvent; + CustomEvent.Type = uint8_t(RecordType::Metadata); + CustomEvent.RecordKind = + uint8_t(MetadataRecord::RecordKinds::CustomEventMarker); + constexpr auto TSCSize = sizeof(std::get<0>(TSC_CPU)); + std::memcpy(&CustomEvent.Data, &ReducedEventSize, sizeof(int32_t)); + std::memcpy(&CustomEvent.Data[sizeof(int32_t)], &TSC, TSCSize); + std::memcpy(RecordPtr, &CustomEvent, sizeof(CustomEvent)); + RecordPtr += sizeof(CustomEvent); + std::memcpy(RecordPtr, Event, ReducedEventSize); + endBufferIfFull(); +} + +XRayLogInitStatus fdrLoggingInit(std::size_t BufferSize, std::size_t BufferMax, + void *Options, + size_t OptionsSize) XRAY_NEVER_INSTRUMENT { + if (OptionsSize != sizeof(FDRLoggingOptions)) + return static_cast<XRayLogInitStatus>(__sanitizer::atomic_load( + &LoggingStatus, __sanitizer::memory_order_acquire)); + s32 CurrentStatus = XRayLogInitStatus::XRAY_LOG_UNINITIALIZED; + if (!__sanitizer::atomic_compare_exchange_strong( + &LoggingStatus, &CurrentStatus, + XRayLogInitStatus::XRAY_LOG_INITIALIZING, + __sanitizer::memory_order_release)) + return static_cast<XRayLogInitStatus>(CurrentStatus); + + { + __sanitizer::SpinMutexLock Guard(&FDROptionsMutex); + memcpy(&FDROptions, Options, OptionsSize); + } + + bool Success = false; + BQ = std::make_shared<BufferQueue>(BufferSize, BufferMax, Success); + if (!Success) { + Report("BufferQueue init failed.\n"); + return XRayLogInitStatus::XRAY_LOG_UNINITIALIZED; + } + + // Install the actual handleArg0 handler after initialising the buffers. + __xray_set_handler(fdrLoggingHandleArg0); + __xray_set_customevent_handler(fdrLoggingHandleCustomEvent); + + __sanitizer::atomic_store(&LoggingStatus, + XRayLogInitStatus::XRAY_LOG_INITIALIZED, + __sanitizer::memory_order_release); + Report("XRay FDR init successful.\n"); + return XRayLogInitStatus::XRAY_LOG_INITIALIZED; +} + +} // namespace __xray + +static auto UNUSED Unused = [] { + using namespace __xray; + if (flags()->xray_fdr_log) { + XRayLogImpl Impl{ + fdrLoggingInit, fdrLoggingFinalize, fdrLoggingHandleArg0, + fdrLoggingFlush, + }; + __xray_set_log_impl(Impl); + } + return true; +}(); diff --git a/contrib/compiler-rt/lib/xray/xray_fdr_logging.h b/contrib/compiler-rt/lib/xray/xray_fdr_logging.h new file mode 100644 index 0000000..426b54d --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_fdr_logging.h @@ -0,0 +1,38 @@ +//===-- xray_fdr_logging.h ------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a function call tracing system. +// +//===----------------------------------------------------------------------===// +#ifndef XRAY_XRAY_FDR_LOGGING_H +#define XRAY_XRAY_FDR_LOGGING_H + +#include "xray/xray_log_interface.h" +#include "xray_fdr_log_records.h" + +// FDR (Flight Data Recorder) Mode +// =============================== +// +// The XRay whitepaper describes a mode of operation for function call trace +// logging that involves writing small records into an in-memory circular +// buffer, that then gets logged to disk on demand. To do this efficiently and +// capture as much data as we can, we use smaller records compared to the +// default mode of always writing fixed-size records. + +namespace __xray { +XRayLogInitStatus fdrLoggingInit(size_t BufferSize, size_t BufferMax, + void *Options, size_t OptionsSize); +XRayLogInitStatus fdrLoggingFinalize(); +void fdrLoggingHandleArg0(int32_t FuncId, XRayEntryType Entry); +XRayLogFlushStatus fdrLoggingFlush(); +XRayLogInitStatus fdrLoggingReset(); + +} // namespace __xray + +#endif // XRAY_XRAY_FDR_LOGGING_H diff --git a/contrib/compiler-rt/lib/xray/xray_fdr_logging_impl.h b/contrib/compiler-rt/lib/xray/xray_fdr_logging_impl.h new file mode 100644 index 0000000..4a1d80f --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_fdr_logging_impl.h @@ -0,0 +1,694 @@ +//===-- xray_fdr_logging_impl.h ---------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Here we implement the thread local state management and record i/o for Flight +// Data Recorder mode for XRay, where we use compact structures to store records +// in memory as well as when writing out the data to files. +// +//===----------------------------------------------------------------------===// +#ifndef XRAY_XRAY_FDR_LOGGING_IMPL_H +#define XRAY_XRAY_FDR_LOGGING_IMPL_H + +#include <cassert> +#include <cstdint> +#include <cstring> +#include <limits> +#include <memory> +#include <string> +#include <sys/syscall.h> +#include <time.h> +#include <unistd.h> + +#include "sanitizer_common/sanitizer_common.h" +#include "xray/xray_log_interface.h" +#include "xray_buffer_queue.h" +#include "xray_defs.h" +#include "xray_fdr_log_records.h" +#include "xray_flags.h" +#include "xray_tsc.h" + +namespace __xray { + +__sanitizer::atomic_sint32_t LoggingStatus = { + XRayLogInitStatus::XRAY_LOG_UNINITIALIZED}; + +/// We expose some of the state transitions when FDR logging mode is operating +/// such that we can simulate a series of log events that may occur without +/// and test with determinism without worrying about the real CPU time. +/// +/// Because the code uses thread_local allocation extensively as part of its +/// design, callers that wish to test events occuring on different threads +/// will actually have to run them on different threads. +/// +/// This also means that it is possible to break invariants maintained by +/// cooperation with xray_fdr_logging class, so be careful and think twice. +namespace __xray_fdr_internal { + +/// Writes the new buffer record and wallclock time that begin a buffer for a +/// thread to MemPtr and increments MemPtr. Bypasses the thread local state +/// machine and writes directly to memory without checks. +static void writeNewBufferPreamble(pid_t Tid, timespec TS, char *&MemPtr); + +/// Write a metadata record to switch to a new CPU to MemPtr and increments +/// MemPtr. Bypasses the thread local state machine and writes directly to +/// memory without checks. +static void writeNewCPUIdMetadata(uint16_t CPU, uint64_t TSC, char *&MemPtr); + +/// Writes an EOB metadata record to MemPtr and increments MemPtr. Bypasses the +/// thread local state machine and writes directly to memory without checks. +static void writeEOBMetadata(char *&MemPtr); + +/// Writes a TSC Wrap metadata record to MemPtr and increments MemPtr. Bypasses +/// the thread local state machine and directly writes to memory without checks. +static void writeTSCWrapMetadata(uint64_t TSC, char *&MemPtr); + +/// Writes a Function Record to MemPtr and increments MemPtr. Bypasses the +/// thread local state machine and writes the function record directly to +/// memory. +static void writeFunctionRecord(int FuncId, uint32_t TSCDelta, + XRayEntryType EntryType, char *&MemPtr); + +/// Sets up a new buffer in thread_local storage and writes a preamble. The +/// wall_clock_reader function is used to populate the WallTimeRecord entry. +static void setupNewBuffer(int (*wall_clock_reader)(clockid_t, + struct timespec *)); + +/// Called to record CPU time for a new CPU within the current thread. +static void writeNewCPUIdMetadata(uint16_t CPU, uint64_t TSC); + +/// Called to close the buffer when the thread exhausts the buffer or when the +/// thread exits (via a thread local variable destructor). +static void writeEOBMetadata(); + +/// TSC Wrap records are written when a TSC delta encoding scheme overflows. +static void writeTSCWrapMetadata(uint64_t TSC); + +/// Here's where the meat of the processing happens. The writer captures +/// function entry, exit and tail exit points with a time and will create +/// TSCWrap, NewCPUId and Function records as necessary. The writer might +/// walk backward through its buffer and erase trivial functions to avoid +/// polluting the log and may use the buffer queue to obtain or release a +/// buffer. +static void processFunctionHook(int32_t FuncId, XRayEntryType Entry, + uint64_t TSC, unsigned char CPU, + int (*wall_clock_reader)(clockid_t, + struct timespec *), + __sanitizer::atomic_sint32_t &LoggingStatus, + const std::shared_ptr<BufferQueue> &BQ); + +//-----------------------------------------------------------------------------| +// The rest of the file is implementation. | +//-----------------------------------------------------------------------------| +// Functions are implemented in the header for inlining since we don't want | +// to grow the stack when we've hijacked the binary for logging. | +//-----------------------------------------------------------------------------| + +namespace { + +thread_local BufferQueue::Buffer Buffer; +thread_local char *RecordPtr = nullptr; + +// The number of FunctionEntry records immediately preceding RecordPtr. +thread_local uint8_t NumConsecutiveFnEnters = 0; + +// The number of adjacent, consecutive pairs of FunctionEntry, Tail Exit +// records preceding RecordPtr. +thread_local uint8_t NumTailCalls = 0; + +constexpr auto MetadataRecSize = sizeof(MetadataRecord); +constexpr auto FunctionRecSize = sizeof(FunctionRecord); + +// We use a thread_local variable to keep track of which CPUs we've already +// run, and the TSC times for these CPUs. This allows us to stop repeating the +// CPU field in the function records. +// +// We assume that we'll support only 65536 CPUs for x86_64. +thread_local uint16_t CurrentCPU = std::numeric_limits<uint16_t>::max(); +thread_local uint64_t LastTSC = 0; +thread_local uint64_t LastFunctionEntryTSC = 0; + +class ThreadExitBufferCleanup { + std::shared_ptr<BufferQueue> &Buffers; + BufferQueue::Buffer &Buffer; + +public: + explicit ThreadExitBufferCleanup(std::shared_ptr<BufferQueue> &BQ, + BufferQueue::Buffer &Buffer) + XRAY_NEVER_INSTRUMENT : Buffers(BQ), + Buffer(Buffer) {} + + ~ThreadExitBufferCleanup() noexcept XRAY_NEVER_INSTRUMENT { + if (RecordPtr == nullptr) + return; + + // We make sure that upon exit, a thread will write out the EOB + // MetadataRecord in the thread-local log, and also release the buffer to + // the queue. + assert((RecordPtr + MetadataRecSize) - static_cast<char *>(Buffer.Buffer) >= + static_cast<ptrdiff_t>(MetadataRecSize)); + if (Buffers) { + writeEOBMetadata(); + auto EC = Buffers->releaseBuffer(Buffer); + if (EC != BufferQueue::ErrorCode::Ok) + Report("Failed to release buffer at %p; error=%s\n", Buffer.Buffer, + BufferQueue::getErrorString(EC)); + Buffers = nullptr; + return; + } + } +}; + +// Make sure a thread that's ever called handleArg0 has a thread-local +// live reference to the buffer queue for this particular instance of +// FDRLogging, and that we're going to clean it up when the thread exits. +thread_local std::shared_ptr<BufferQueue> LocalBQ = nullptr; +thread_local ThreadExitBufferCleanup Cleanup(LocalBQ, Buffer); + +class RecursionGuard { + bool &Running; + const bool Valid; + +public: + explicit RecursionGuard(bool &R) : Running(R), Valid(!R) { + if (Valid) + Running = true; + } + + RecursionGuard(const RecursionGuard &) = delete; + RecursionGuard(RecursionGuard &&) = delete; + RecursionGuard &operator=(const RecursionGuard &) = delete; + RecursionGuard &operator=(RecursionGuard &&) = delete; + + explicit operator bool() const { return Valid; } + + ~RecursionGuard() noexcept { + if (Valid) + Running = false; + } +}; + +inline bool loggingInitialized( + const __sanitizer::atomic_sint32_t &LoggingStatus) XRAY_NEVER_INSTRUMENT { + return __sanitizer::atomic_load(&LoggingStatus, + __sanitizer::memory_order_acquire) == + XRayLogInitStatus::XRAY_LOG_INITIALIZED; +} + +} // namespace + +inline void writeNewBufferPreamble(pid_t Tid, timespec TS, + char *&MemPtr) XRAY_NEVER_INSTRUMENT { + static constexpr int InitRecordsCount = 2; + std::aligned_storage<sizeof(MetadataRecord)>::type Records[InitRecordsCount]; + { + // Write out a MetadataRecord to signify that this is the start of a new + // buffer, associated with a particular thread, with a new CPU. For the + // data, we have 15 bytes to squeeze as much information as we can. At this + // point we only write down the following bytes: + // - Thread ID (pid_t, 4 bytes) + auto &NewBuffer = *reinterpret_cast<MetadataRecord *>(&Records[0]); + NewBuffer.Type = uint8_t(RecordType::Metadata); + NewBuffer.RecordKind = uint8_t(MetadataRecord::RecordKinds::NewBuffer); + std::memcpy(&NewBuffer.Data, &Tid, sizeof(pid_t)); + } + // Also write the WalltimeMarker record. + { + static_assert(sizeof(time_t) <= 8, "time_t needs to be at most 8 bytes"); + auto &WalltimeMarker = *reinterpret_cast<MetadataRecord *>(&Records[1]); + WalltimeMarker.Type = uint8_t(RecordType::Metadata); + WalltimeMarker.RecordKind = + uint8_t(MetadataRecord::RecordKinds::WalltimeMarker); + + // We only really need microsecond precision here, and enforce across + // platforms that we need 64-bit seconds and 32-bit microseconds encoded in + // the Metadata record. + int32_t Micros = TS.tv_nsec / 1000; + int64_t Seconds = TS.tv_sec; + std::memcpy(WalltimeMarker.Data, &Seconds, sizeof(Seconds)); + std::memcpy(WalltimeMarker.Data + sizeof(Seconds), &Micros, sizeof(Micros)); + } + std::memcpy(MemPtr, Records, sizeof(MetadataRecord) * InitRecordsCount); + MemPtr += sizeof(MetadataRecord) * InitRecordsCount; + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; +} + +inline void setupNewBuffer(int (*wall_clock_reader)( + clockid_t, struct timespec *)) XRAY_NEVER_INSTRUMENT { + RecordPtr = static_cast<char *>(Buffer.Buffer); + pid_t Tid = syscall(SYS_gettid); + timespec TS{0, 0}; + // This is typically clock_gettime, but callers have injection ability. + wall_clock_reader(CLOCK_MONOTONIC, &TS); + writeNewBufferPreamble(Tid, TS, RecordPtr); + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; +} + +inline void writeNewCPUIdMetadata(uint16_t CPU, uint64_t TSC, + char *&MemPtr) XRAY_NEVER_INSTRUMENT { + MetadataRecord NewCPUId; + NewCPUId.Type = uint8_t(RecordType::Metadata); + NewCPUId.RecordKind = uint8_t(MetadataRecord::RecordKinds::NewCPUId); + + // The data for the New CPU will contain the following bytes: + // - CPU ID (uint16_t, 2 bytes) + // - Full TSC (uint64_t, 8 bytes) + // Total = 10 bytes. + std::memcpy(&NewCPUId.Data, &CPU, sizeof(CPU)); + std::memcpy(&NewCPUId.Data[sizeof(CPU)], &TSC, sizeof(TSC)); + std::memcpy(MemPtr, &NewCPUId, sizeof(MetadataRecord)); + MemPtr += sizeof(MetadataRecord); + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; +} + +inline void writeNewCPUIdMetadata(uint16_t CPU, + uint64_t TSC) XRAY_NEVER_INSTRUMENT { + writeNewCPUIdMetadata(CPU, TSC, RecordPtr); +} + +inline void writeEOBMetadata(char *&MemPtr) XRAY_NEVER_INSTRUMENT { + MetadataRecord EOBMeta; + EOBMeta.Type = uint8_t(RecordType::Metadata); + EOBMeta.RecordKind = uint8_t(MetadataRecord::RecordKinds::EndOfBuffer); + // For now we don't write any bytes into the Data field. + std::memcpy(MemPtr, &EOBMeta, sizeof(MetadataRecord)); + MemPtr += sizeof(MetadataRecord); + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; +} + +inline void writeEOBMetadata() XRAY_NEVER_INSTRUMENT { + writeEOBMetadata(RecordPtr); +} + +inline void writeTSCWrapMetadata(uint64_t TSC, + char *&MemPtr) XRAY_NEVER_INSTRUMENT { + MetadataRecord TSCWrap; + TSCWrap.Type = uint8_t(RecordType::Metadata); + TSCWrap.RecordKind = uint8_t(MetadataRecord::RecordKinds::TSCWrap); + + // The data for the TSCWrap record contains the following bytes: + // - Full TSC (uint64_t, 8 bytes) + // Total = 8 bytes. + std::memcpy(&TSCWrap.Data, &TSC, sizeof(TSC)); + std::memcpy(MemPtr, &TSCWrap, sizeof(MetadataRecord)); + MemPtr += sizeof(MetadataRecord); + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; +} + +inline void writeTSCWrapMetadata(uint64_t TSC) XRAY_NEVER_INSTRUMENT { + writeTSCWrapMetadata(TSC, RecordPtr); +} + +inline void writeFunctionRecord(int FuncId, uint32_t TSCDelta, + XRayEntryType EntryType, + char *&MemPtr) XRAY_NEVER_INSTRUMENT { + std::aligned_storage<sizeof(FunctionRecord), alignof(FunctionRecord)>::type + AlignedFuncRecordBuffer; + auto &FuncRecord = + *reinterpret_cast<FunctionRecord *>(&AlignedFuncRecordBuffer); + FuncRecord.Type = uint8_t(RecordType::Function); + // Only take 28 bits of the function id. + FuncRecord.FuncId = FuncId & ~(0x0F << 28); + FuncRecord.TSCDelta = TSCDelta; + + switch (EntryType) { + case XRayEntryType::ENTRY: + ++NumConsecutiveFnEnters; + FuncRecord.RecordKind = uint8_t(FunctionRecord::RecordKinds::FunctionEnter); + break; + case XRayEntryType::LOG_ARGS_ENTRY: + // We should not rewind functions with logged args. + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; + FuncRecord.RecordKind = uint8_t(FunctionRecord::RecordKinds::FunctionEnter); + break; + case XRayEntryType::EXIT: + // If we've decided to log the function exit, we will never erase the log + // before it. + NumConsecutiveFnEnters = 0; + NumTailCalls = 0; + FuncRecord.RecordKind = uint8_t(FunctionRecord::RecordKinds::FunctionExit); + break; + case XRayEntryType::TAIL: + // If we just entered the function we're tail exiting from or erased every + // invocation since then, this function entry tail pair is a candidate to + // be erased when the child function exits. + if (NumConsecutiveFnEnters > 0) { + ++NumTailCalls; + NumConsecutiveFnEnters = 0; + } else { + // We will never be able to erase this tail call since we have logged + // something in between the function entry and tail exit. + NumTailCalls = 0; + NumConsecutiveFnEnters = 0; + } + FuncRecord.RecordKind = + uint8_t(FunctionRecord::RecordKinds::FunctionTailExit); + break; + case XRayEntryType::CUSTOM_EVENT: { + // This is a bug in patching, so we'll report it once and move on. + static bool Once = [&] { + Report("Internal error: patched an XRay custom event call as a function; " + "func id = %d\n", + FuncId); + return true; + }(); + (void)Once; + return; + } + } + + std::memcpy(MemPtr, &AlignedFuncRecordBuffer, sizeof(FunctionRecord)); + MemPtr += sizeof(FunctionRecord); +} + +static uint64_t thresholdTicks() { + static uint64_t TicksPerSec = probeRequiredCPUFeatures() + ? getTSCFrequency() + : __xray::NanosecondsPerSecond; + static const uint64_t ThresholdTicks = + TicksPerSec * flags()->xray_fdr_log_func_duration_threshold_us / 1000000; + return ThresholdTicks; +} + +// Re-point the thread local pointer into this thread's Buffer before the recent +// "Function Entry" record and any "Tail Call Exit" records after that. +static void rewindRecentCall(uint64_t TSC, uint64_t &LastTSC, + uint64_t &LastFunctionEntryTSC, int32_t FuncId) { + using AlignedFuncStorage = + std::aligned_storage<sizeof(FunctionRecord), + alignof(FunctionRecord)>::type; + RecordPtr -= FunctionRecSize; + AlignedFuncStorage AlignedFuncRecordBuffer; + const auto &FuncRecord = *reinterpret_cast<FunctionRecord *>( + std::memcpy(&AlignedFuncRecordBuffer, RecordPtr, FunctionRecSize)); + assert(FuncRecord.RecordKind == + uint8_t(FunctionRecord::RecordKinds::FunctionEnter) && + "Expected to find function entry recording when rewinding."); + assert(FuncRecord.FuncId == (FuncId & ~(0x0F << 28)) && + "Expected matching function id when rewinding Exit"); + --NumConsecutiveFnEnters; + LastTSC -= FuncRecord.TSCDelta; + + // We unwound one call. Update the state and return without writing a log. + if (NumConsecutiveFnEnters != 0) { + LastFunctionEntryTSC -= FuncRecord.TSCDelta; + return; + } + + // Otherwise we've rewound the stack of all function entries, we might be + // able to rewind further by erasing tail call functions that are being + // exited from via this exit. + LastFunctionEntryTSC = 0; + auto RewindingTSC = LastTSC; + auto RewindingRecordPtr = RecordPtr - FunctionRecSize; + while (NumTailCalls > 0) { + AlignedFuncStorage TailExitRecordBuffer; + // Rewind the TSC back over the TAIL EXIT record. + const auto &ExpectedTailExit = + *reinterpret_cast<FunctionRecord *>(std::memcpy( + &TailExitRecordBuffer, RewindingRecordPtr, FunctionRecSize)); + + assert(ExpectedTailExit.RecordKind == + uint8_t(FunctionRecord::RecordKinds::FunctionTailExit) && + "Expected to find tail exit when rewinding."); + RewindingRecordPtr -= FunctionRecSize; + RewindingTSC -= ExpectedTailExit.TSCDelta; + AlignedFuncStorage FunctionEntryBuffer; + const auto &ExpectedFunctionEntry = *reinterpret_cast<FunctionRecord *>( + std::memcpy(&FunctionEntryBuffer, RewindingRecordPtr, FunctionRecSize)); + assert(ExpectedFunctionEntry.RecordKind == + uint8_t(FunctionRecord::RecordKinds::FunctionEnter) && + "Expected to find function entry when rewinding tail call."); + assert(ExpectedFunctionEntry.FuncId == ExpectedTailExit.FuncId && + "Expected funcids to match when rewinding tail call."); + + // This tail call exceeded the threshold duration. It will not be erased. + if ((TSC - RewindingTSC) >= thresholdTicks()) { + NumTailCalls = 0; + return; + } + + // We can erase a tail exit pair that we're exiting through since + // its duration is under threshold. + --NumTailCalls; + RewindingRecordPtr -= FunctionRecSize; + RewindingTSC -= ExpectedFunctionEntry.TSCDelta; + RecordPtr -= 2 * FunctionRecSize; + LastTSC = RewindingTSC; + } +} + +inline bool releaseThreadLocalBuffer(BufferQueue *BQ) { + auto EC = BQ->releaseBuffer(Buffer); + if (EC != BufferQueue::ErrorCode::Ok) { + Report("Failed to release buffer at %p; error=%s\n", Buffer.Buffer, + BufferQueue::getErrorString(EC)); + return false; + } + return true; +} + +inline bool prepareBuffer(int (*wall_clock_reader)(clockid_t, + struct timespec *), + size_t MaxSize) XRAY_NEVER_INSTRUMENT { + char *BufferStart = static_cast<char *>(Buffer.Buffer); + if ((RecordPtr + MaxSize) > (BufferStart + Buffer.Size - MetadataRecSize)) { + writeEOBMetadata(); + if (!releaseThreadLocalBuffer(LocalBQ.get())) + return false; + auto EC = LocalBQ->getBuffer(Buffer); + if (EC != BufferQueue::ErrorCode::Ok) { + Report("Failed to acquire a buffer; error=%s\n", + BufferQueue::getErrorString(EC)); + return false; + } + setupNewBuffer(wall_clock_reader); + } + return true; +} + +inline bool isLogInitializedAndReady( + std::shared_ptr<BufferQueue> &LocalBQ, uint64_t TSC, unsigned char CPU, + int (*wall_clock_reader)(clockid_t, + struct timespec *)) XRAY_NEVER_INSTRUMENT { + // Bail out right away if logging is not initialized yet. + // We should take the opportunity to release the buffer though. + auto Status = __sanitizer::atomic_load(&LoggingStatus, + __sanitizer::memory_order_acquire); + if (Status != XRayLogInitStatus::XRAY_LOG_INITIALIZED) { + if (RecordPtr != nullptr && + (Status == XRayLogInitStatus::XRAY_LOG_FINALIZING || + Status == XRayLogInitStatus::XRAY_LOG_FINALIZED)) { + writeEOBMetadata(); + if (!releaseThreadLocalBuffer(LocalBQ.get())) + return false; + RecordPtr = nullptr; + LocalBQ = nullptr; + return false; + } + return false; + } + + if (!loggingInitialized(LoggingStatus) || LocalBQ->finalizing()) { + writeEOBMetadata(); + if (!releaseThreadLocalBuffer(LocalBQ.get())) + return false; + RecordPtr = nullptr; + } + + if (Buffer.Buffer == nullptr) { + auto EC = LocalBQ->getBuffer(Buffer); + if (EC != BufferQueue::ErrorCode::Ok) { + auto LS = __sanitizer::atomic_load(&LoggingStatus, + __sanitizer::memory_order_acquire); + if (LS != XRayLogInitStatus::XRAY_LOG_FINALIZING && + LS != XRayLogInitStatus::XRAY_LOG_FINALIZED) + Report("Failed to acquire a buffer; error=%s\n", + BufferQueue::getErrorString(EC)); + return false; + } + + setupNewBuffer(wall_clock_reader); + } + + if (CurrentCPU == std::numeric_limits<uint16_t>::max()) { + // This means this is the first CPU this thread has ever run on. We set + // the current CPU and record this as the first TSC we've seen. + CurrentCPU = CPU; + writeNewCPUIdMetadata(CPU, TSC); + } + + return true; +} // namespace __xray_fdr_internal + +inline void endBufferIfFull() XRAY_NEVER_INSTRUMENT { + auto BufferStart = static_cast<char *>(Buffer.Buffer); + if ((RecordPtr + MetadataRecSize) - BufferStart == MetadataRecSize) { + writeEOBMetadata(); + if (!releaseThreadLocalBuffer(LocalBQ.get())) + return; + RecordPtr = nullptr; + } +} + +inline void processFunctionHook( + int32_t FuncId, XRayEntryType Entry, uint64_t TSC, unsigned char CPU, + int (*wall_clock_reader)(clockid_t, struct timespec *), + __sanitizer::atomic_sint32_t &LoggingStatus, + const std::shared_ptr<BufferQueue> &BQ) XRAY_NEVER_INSTRUMENT { + // Prevent signal handler recursion, so in case we're already in a log writing + // mode and the signal handler comes in (and is also instrumented) then we + // don't want to be clobbering potentially partial writes already happening in + // the thread. We use a simple thread_local latch to only allow one on-going + // handleArg0 to happen at any given time. + thread_local bool Running = false; + RecursionGuard Guard{Running}; + if (!Guard) { + assert(Running == true && "RecursionGuard is buggy!"); + return; + } + + // In case the reference has been cleaned up before, we make sure we + // initialize it to the provided BufferQueue. + if (LocalBQ == nullptr) + LocalBQ = BQ; + + if (!isLogInitializedAndReady(LocalBQ, TSC, CPU, wall_clock_reader)) + return; + + // Before we go setting up writing new function entries, we need to be really + // careful about the pointer math we're doing. This means we need to ensure + // that the record we are about to write is going to fit into the buffer, + // without overflowing the buffer. + // + // To do this properly, we use the following assumptions: + // + // - The least number of bytes we will ever write is 8 + // (sizeof(FunctionRecord)) only if the delta between the previous entry + // and this entry is within 32 bits. + // - The most number of bytes we will ever write is 8 + 16 = 24. This is + // computed by: + // + // sizeof(FunctionRecord) + sizeof(MetadataRecord) + // + // These arise in the following cases: + // + // 1. When the delta between the TSC we get and the previous TSC for the + // same CPU is outside of the uint32_t range, we end up having to + // write a MetadataRecord to indicate a "tsc wrap" before the actual + // FunctionRecord. + // 2. When we learn that we've moved CPUs, we need to write a + // MetadataRecord to indicate a "cpu change", and thus write out the + // current TSC for that CPU before writing out the actual + // FunctionRecord. + // 3. When we learn about a new CPU ID, we need to write down a "new cpu + // id" MetadataRecord before writing out the actual FunctionRecord. + // + // - An End-of-Buffer (EOB) MetadataRecord is 16 bytes. + // + // So the math we need to do is to determine whether writing 24 bytes past the + // current pointer leaves us with enough bytes to write the EOB + // MetadataRecord. If we don't have enough space after writing as much as 24 + // bytes in the end of the buffer, we need to write out the EOB, get a new + // Buffer, set it up properly before doing any further writing. + // + if (!prepareBuffer(wall_clock_reader, FunctionRecSize + MetadataRecSize)) { + LocalBQ = nullptr; + return; + } + + // By this point, we are now ready to write at most 24 bytes (one metadata + // record and one function record). + assert((RecordPtr + (MetadataRecSize + FunctionRecSize)) - + static_cast<char *>(Buffer.Buffer) >= + static_cast<ptrdiff_t>(MetadataRecSize) && + "Misconfigured BufferQueue provided; Buffer size not large enough."); + + // Here we compute the TSC Delta. There are a few interesting situations we + // need to account for: + // + // - The thread has migrated to a different CPU. If this is the case, then + // we write down the following records: + // + // 1. A 'NewCPUId' Metadata record. + // 2. A FunctionRecord with a 0 for the TSCDelta field. + // + // - The TSC delta is greater than the 32 bits we can store in a + // FunctionRecord. In this case we write down the following records: + // + // 1. A 'TSCWrap' Metadata record. + // 2. A FunctionRecord with a 0 for the TSCDelta field. + // + // - The TSC delta is representable within the 32 bits we can store in a + // FunctionRecord. In this case we write down just a FunctionRecord with + // the correct TSC delta. + // + uint32_t RecordTSCDelta = 0; + if (CPU != CurrentCPU) { + // We've moved to a new CPU. + writeNewCPUIdMetadata(CPU, TSC); + } else { + // If the delta is greater than the range for a uint32_t, then we write out + // the TSC wrap metadata entry with the full TSC, and the TSC for the + // function record be 0. + auto Delta = TSC - LastTSC; + if (Delta > (1ULL << 32) - 1) + writeTSCWrapMetadata(TSC); + else + RecordTSCDelta = Delta; + } + + LastTSC = TSC; + CurrentCPU = CPU; + switch (Entry) { + case XRayEntryType::ENTRY: + case XRayEntryType::LOG_ARGS_ENTRY: + // Update the thread local state for the next invocation. + LastFunctionEntryTSC = TSC; + break; + case XRayEntryType::TAIL: + break; + case XRayEntryType::EXIT: + // Break out and write the exit record if we can't erase any functions. + if (NumConsecutiveFnEnters == 0 || + (TSC - LastFunctionEntryTSC) >= thresholdTicks()) + break; + rewindRecentCall(TSC, LastTSC, LastFunctionEntryTSC, FuncId); + return; // without writing log. + case XRayEntryType::CUSTOM_EVENT: { + // This is a bug in patching, so we'll report it once and move on. + static bool Once = [&] { + Report("Internal error: patched an XRay custom event call as a function; " + "func id = %d", + FuncId); + return true; + }(); + (void)Once; + return; + } + } + + writeFunctionRecord(FuncId, RecordTSCDelta, Entry, RecordPtr); + + // If we've exhausted the buffer by this time, we then release the buffer to + // make sure that other threads may start using this buffer. + endBufferIfFull(); +} + +} // namespace __xray_fdr_internal +} // namespace __xray + +#endif // XRAY_XRAY_FDR_LOGGING_IMPL_H diff --git a/contrib/compiler-rt/lib/xray/xray_flags.cc b/contrib/compiler-rt/lib/xray/xray_flags.cc index 338c237..1ee4d10 100644 --- a/contrib/compiler-rt/lib/xray/xray_flags.cc +++ b/contrib/compiler-rt/lib/xray/xray_flags.cc @@ -24,31 +24,55 @@ namespace __xray { Flags xray_flags_dont_use_directly; // use via flags(). -void Flags::SetDefaults() XRAY_NEVER_INSTRUMENT { +void Flags::setDefaults() XRAY_NEVER_INSTRUMENT { #define XRAY_FLAG(Type, Name, DefaultValue, Description) Name = DefaultValue; #include "xray_flags.inc" #undef XRAY_FLAG } -static void RegisterXRayFlags(FlagParser *P, Flags *F) XRAY_NEVER_INSTRUMENT { +static void registerXRayFlags(FlagParser *P, Flags *F) XRAY_NEVER_INSTRUMENT { #define XRAY_FLAG(Type, Name, DefaultValue, Description) \ RegisterFlag(P, #Name, Description, &F->Name); #include "xray_flags.inc" #undef XRAY_FLAG } -void InitializeFlags() XRAY_NEVER_INSTRUMENT { +// This function, as defined with the help of a macro meant to be introduced at +// build time of the XRay runtime, passes in a statically defined list of +// options that control XRay. This means users/deployments can tweak the +// defaults that override the hard-coded defaults in the xray_flags.inc at +// compile-time using the XRAY_DEFAULT_OPTIONS macro. +static const char *useCompilerDefinedFlags() XRAY_NEVER_INSTRUMENT { +#ifdef XRAY_DEFAULT_OPTIONS +// Do the double-layered string conversion to prevent badly crafted strings +// provided through the XRAY_DEFAULT_OPTIONS from causing compilation issues (or +// changing the semantics of the implementation through the macro). This ensures +// that we convert whatever XRAY_DEFAULT_OPTIONS is defined as a string literal. +#define XRAY_STRINGIZE(x) #x +#define XRAY_STRINGIZE_OPTIONS(options) XRAY_STRINGIZE(options) + return XRAY_STRINGIZE_OPTIONS(XRAY_DEFAULT_OPTIONS); +#else + return ""; +#endif +} + +void initializeFlags() XRAY_NEVER_INSTRUMENT { SetCommonFlagsDefaults(); auto *F = flags(); - F->SetDefaults(); + F->setDefaults(); FlagParser XRayParser; - RegisterXRayFlags(&XRayParser, F); + registerXRayFlags(&XRayParser, F); RegisterCommonFlags(&XRayParser); - // Override from command line. + // Use options defaulted at compile-time for the runtime. + const char *XRayCompileFlags = useCompilerDefinedFlags(); + XRayParser.ParseString(XRayCompileFlags); + + // Override from environment variables. XRayParser.ParseString(GetEnv("XRAY_OPTIONS")); + // Override from command line. InitializeCommonFlags(); if (Verbosity()) diff --git a/contrib/compiler-rt/lib/xray/xray_flags.h b/contrib/compiler-rt/lib/xray/xray_flags.h index 2ecf5fb..f4e3028 100644 --- a/contrib/compiler-rt/lib/xray/xray_flags.h +++ b/contrib/compiler-rt/lib/xray/xray_flags.h @@ -24,13 +24,13 @@ struct Flags { #include "xray_flags.inc" #undef XRAY_FLAG - void SetDefaults(); + void setDefaults(); }; extern Flags xray_flags_dont_use_directly; inline Flags *flags() { return &xray_flags_dont_use_directly; } -void InitializeFlags(); +void initializeFlags(); } // namespace __xray diff --git a/contrib/compiler-rt/lib/xray/xray_flags.inc b/contrib/compiler-rt/lib/xray/xray_flags.inc index 0f6ced8..7ddce78 100644 --- a/contrib/compiler-rt/lib/xray/xray_flags.inc +++ b/contrib/compiler-rt/lib/xray/xray_flags.inc @@ -14,9 +14,14 @@ #error "Define XRAY_FLAG prior to including this file!" #endif -XRAY_FLAG(bool, patch_premain, true, +XRAY_FLAG(bool, patch_premain, false, "Whether to patch instrumentation points before main.") XRAY_FLAG(bool, xray_naive_log, true, "Whether to install the naive log implementation.") XRAY_FLAG(const char *, xray_logfile_base, "xray-log.", "Filename base for the xray logfile.") +XRAY_FLAG(bool, xray_fdr_log, false, + "Whether to install the flight data recorder logging implementation.") +XRAY_FLAG(int, xray_fdr_log_func_duration_threshold_us, 5, + "FDR logging will try to skip functions that execute for fewer " + "microseconds than this threshold.") diff --git a/contrib/compiler-rt/lib/xray/xray_init.cc b/contrib/compiler-rt/lib/xray/xray_init.cc index eb86182..aa660ba 100644 --- a/contrib/compiler-rt/lib/xray/xray_init.cc +++ b/contrib/compiler-rt/lib/xray/xray_init.cc @@ -12,7 +12,6 @@ // XRay initialisation logic. //===----------------------------------------------------------------------===// -#include <atomic> #include <fcntl.h> #include <strings.h> #include <unistd.h> @@ -26,9 +25,10 @@ extern "C" { void __xray_init(); extern const XRaySledEntry __start_xray_instr_map[] __attribute__((weak)); extern const XRaySledEntry __stop_xray_instr_map[] __attribute__((weak)); +extern const XRayFunctionSledIndex __start_xray_fn_idx[] __attribute__((weak)); +extern const XRayFunctionSledIndex __stop_xray_fn_idx[] __attribute__((weak)); } -using namespace __sanitizer; using namespace __xray; // When set to 'true' this means the XRay runtime has been initialised. We use @@ -38,29 +38,30 @@ using namespace __xray; // // FIXME: Support DSO instrumentation maps too. The current solution only works // for statically linked executables. -std::atomic<bool> XRayInitialized{false}; +__sanitizer::atomic_uint8_t XRayInitialized{0}; // This should always be updated before XRayInitialized is updated. -std::atomic<__xray::XRaySledMap> XRayInstrMap{}; +__sanitizer::SpinMutex XRayInstrMapMutex; +XRaySledMap XRayInstrMap; // __xray_init() will do the actual loading of the current process' memory map // and then proceed to look for the .xray_instr_map section/segment. void __xray_init() XRAY_NEVER_INSTRUMENT { - InitializeFlags(); + initializeFlags(); if (__start_xray_instr_map == nullptr) { Report("XRay instrumentation map missing. Not initializing XRay.\n"); return; } - // Now initialize the XRayInstrMap global struct with the address of the - // entries, reinterpreted as an array of XRaySledEntry objects. We use the - // virtual pointer we have from the section to provide us the correct - // information. - __xray::XRaySledMap SledMap{}; - SledMap.Sleds = __start_xray_instr_map; - SledMap.Entries = __stop_xray_instr_map - __start_xray_instr_map; - XRayInstrMap.store(SledMap, std::memory_order_release); - XRayInitialized.store(true, std::memory_order_release); + { + __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); + XRayInstrMap.Sleds = __start_xray_instr_map; + XRayInstrMap.Entries = __stop_xray_instr_map - __start_xray_instr_map; + XRayInstrMap.SledsIndex = __start_xray_fn_idx; + XRayInstrMap.Functions = __stop_xray_fn_idx - __start_xray_fn_idx; + } + __sanitizer::atomic_store(&XRayInitialized, true, + __sanitizer::memory_order_release); if (flags()->patch_premain) __xray_patch(); diff --git a/contrib/compiler-rt/lib/xray/xray_inmemory_log.cc b/contrib/compiler-rt/lib/xray/xray_inmemory_log.cc index adcb216..83aecfa 100644 --- a/contrib/compiler-rt/lib/xray/xray_inmemory_log.cc +++ b/contrib/compiler-rt/lib/xray/xray_inmemory_log.cc @@ -16,8 +16,6 @@ //===----------------------------------------------------------------------===// #include <cassert> -#include <cstdint> -#include <cstdio> #include <fcntl.h> #include <mutex> #include <sys/stat.h> @@ -26,19 +24,13 @@ #include <thread> #include <unistd.h> -#if defined(__x86_64__) -#include "xray_x86_64.h" -#elif defined(__arm__) || defined(__aarch64__) -#include "xray_emulate_tsc.h" -#else -#error "Unsupported CPU Architecture" -#endif /* Architecture-specific inline intrinsics */ - #include "sanitizer_common/sanitizer_libc.h" #include "xray/xray_records.h" #include "xray_defs.h" #include "xray_flags.h" #include "xray_interface_internal.h" +#include "xray_tsc.h" +#include "xray_utils.h" // __xray_InMemoryRawLog will use a thread-local aligned buffer capped to a // certain size (32kb by default) and use it as if it were a circular buffer for @@ -53,25 +45,6 @@ namespace __xray { std::mutex LogMutex; -static void retryingWriteAll(int Fd, char *Begin, - char *End) XRAY_NEVER_INSTRUMENT { - if (Begin == End) - return; - auto TotalBytes = std::distance(Begin, End); - while (auto Written = write(Fd, Begin, TotalBytes)) { - if (Written < 0) { - if (errno == EINTR) - continue; // Try again. - Report("Failed to write; errno = %d\n", errno); - return; - } - TotalBytes -= Written; - if (TotalBytes == 0) - break; - Begin += Written; - } -} - class ThreadExitFlusher { int Fd; XRayRecord *Start; @@ -102,41 +75,15 @@ public: using namespace __xray; -void PrintToStdErr(const char *Buffer) XRAY_NEVER_INSTRUMENT { - fprintf(stderr, "%s", Buffer); -} - static int __xray_OpenLogFile() XRAY_NEVER_INSTRUMENT { - // FIXME: Figure out how to make this less stderr-dependent. - SetPrintfAndReportCallback(PrintToStdErr); - // Open a temporary file once for the log. - static char TmpFilename[256] = {}; - static char TmpWildcardPattern[] = "XXXXXX"; - auto Argv = GetArgv(); - const char *Progname = Argv[0] == nullptr ? "(unknown)" : Argv[0]; - const char *LastSlash = internal_strrchr(Progname, '/'); - - if (LastSlash != nullptr) - Progname = LastSlash + 1; - - const int HalfLength = sizeof(TmpFilename) / 2 - sizeof(TmpWildcardPattern); - int NeededLength = internal_snprintf(TmpFilename, sizeof(TmpFilename), - "%.*s%.*s.%s", - HalfLength, flags()->xray_logfile_base, - HalfLength, Progname, - TmpWildcardPattern); - if (NeededLength > int(sizeof(TmpFilename))) { - Report("XRay log file name too long (%d): %s\n", NeededLength, TmpFilename); - return -1; - } - int Fd = mkstemp(TmpFilename); - if (Fd == -1) { - Report("XRay: Failed opening temporary file '%s'; not logging events.\n", - TmpFilename); + int F = getLogFD(); + if (F == -1) return -1; - } - if (Verbosity()) - fprintf(stderr, "XRay: Log file in '%s'\n", TmpFilename); + + // Test for required CPU features and cache the cycle frequency + static bool TSCSupported = probeRequiredCPUFeatures(); + static uint64_t CycleFrequency = TSCSupported ? getTSCFrequency() + : __xray::NanosecondsPerSecond; // Since we're here, we get to write the header. We set it up so that the // header will only be written once, at the start, and let the threads @@ -144,19 +91,20 @@ static int __xray_OpenLogFile() XRAY_NEVER_INSTRUMENT { XRayFileHeader Header; Header.Version = 1; Header.Type = FileTypes::NAIVE_LOG; - Header.CycleFrequency = __xray::cycleFrequency(); + Header.CycleFrequency = CycleFrequency; // FIXME: Actually check whether we have 'constant_tsc' and 'nonstop_tsc' // before setting the values in the header. Header.ConstantTSC = 1; Header.NonstopTSC = 1; - retryingWriteAll(Fd, reinterpret_cast<char *>(&Header), + retryingWriteAll(F, reinterpret_cast<char *>(&Header), reinterpret_cast<char *>(&Header) + sizeof(Header)); - return Fd; + return F; } -void __xray_InMemoryRawLog(int32_t FuncId, - XRayEntryType Type) XRAY_NEVER_INSTRUMENT { +template <class RDTSC> +void __xray_InMemoryRawLog(int32_t FuncId, XRayEntryType Type, + RDTSC ReadTSC) XRAY_NEVER_INSTRUMENT { using Buffer = std::aligned_storage<sizeof(XRayRecord), alignof(XRayRecord)>::type; static constexpr size_t BuffLen = 1024; @@ -173,7 +121,7 @@ void __xray_InMemoryRawLog(int32_t FuncId, // through a pointer offset. auto &R = reinterpret_cast<__xray::XRayRecord *>(InMemoryBuffer)[Offset]; R.RecordType = RecordTypes::NORMAL; - R.TSC = __xray::readTSC(R.CPU); + R.TSC = ReadTSC(R.CPU); R.TId = TId; R.Type = Type; R.FuncId = FuncId; @@ -187,8 +135,32 @@ void __xray_InMemoryRawLog(int32_t FuncId, } } -static auto Unused = [] { +void __xray_InMemoryRawLogRealTSC(int32_t FuncId, + XRayEntryType Type) XRAY_NEVER_INSTRUMENT { + __xray_InMemoryRawLog(FuncId, Type, __xray::readTSC); +} + +void __xray_InMemoryEmulateTSC(int32_t FuncId, + XRayEntryType Type) XRAY_NEVER_INSTRUMENT { + __xray_InMemoryRawLog(FuncId, Type, [](uint8_t &CPU) XRAY_NEVER_INSTRUMENT { + timespec TS; + int result = clock_gettime(CLOCK_REALTIME, &TS); + if (result != 0) { + Report("clock_gettimg(2) return %d, errno=%d.", result, int(errno)); + TS = {0, 0}; + } + CPU = 0; + return TS.tv_sec * __xray::NanosecondsPerSecond + TS.tv_nsec; + }); +} + +static auto UNUSED Unused = [] { + auto UseRealTSC = probeRequiredCPUFeatures(); + if (!UseRealTSC) + Report("WARNING: Required CPU features missing for XRay instrumentation, " + "using emulation instead.\n"); if (flags()->xray_naive_log) - __xray_set_handler(__xray_InMemoryRawLog); + __xray_set_handler(UseRealTSC ? __xray_InMemoryRawLogRealTSC + : __xray_InMemoryEmulateTSC); return true; }(); diff --git a/contrib/compiler-rt/lib/xray/xray_interface.cc b/contrib/compiler-rt/lib/xray/xray_interface.cc index 20a2b66..694d34c 100644 --- a/contrib/compiler-rt/lib/xray/xray_interface.cc +++ b/contrib/compiler-rt/lib/xray/xray_interface.cc @@ -15,7 +15,6 @@ #include "xray_interface_internal.h" -#include <atomic> #include <cstdint> #include <cstdio> #include <errno.h> @@ -35,12 +34,24 @@ static const int16_t cSledLength = 12; static const int16_t cSledLength = 32; #elif defined(__arm__) static const int16_t cSledLength = 28; +#elif SANITIZER_MIPS32 +static const int16_t cSledLength = 48; +#elif SANITIZER_MIPS64 +static const int16_t cSledLength = 64; +#elif defined(__powerpc64__) +static const int16_t cSledLength = 8; #else #error "Unsupported CPU Architecture" #endif /* CPU architecture */ // This is the function to call when we encounter the entry or exit sleds. -std::atomic<void (*)(int32_t, XRayEntryType)> XRayPatchedFunction{nullptr}; +__sanitizer::atomic_uintptr_t XRayPatchedFunction{0}; + +// This is the function to call from the arg1-enabled sleds/trampolines. +__sanitizer::atomic_uintptr_t XRayArgLogger{0}; + +// This is the function to call when we encounter a custom event log call. +__sanitizer::atomic_uintptr_t XRayPatchedCustomEvent{0}; // MProtectHelper is an RAII wrapper for calls to mprotect(...) that will undo // any successful mprotect(...) changes. This is used to make a page writeable @@ -79,23 +90,45 @@ public: } // namespace __xray -extern std::atomic<bool> XRayInitialized; -extern std::atomic<__xray::XRaySledMap> XRayInstrMap; +extern __sanitizer::SpinMutex XRayInstrMapMutex; +extern __sanitizer::atomic_uint8_t XRayInitialized; +extern __xray::XRaySledMap XRayInstrMap; int __xray_set_handler(void (*entry)(int32_t, XRayEntryType)) XRAY_NEVER_INSTRUMENT { - if (XRayInitialized.load(std::memory_order_acquire)) { - __xray::XRayPatchedFunction.store(entry, std::memory_order_release); + if (__sanitizer::atomic_load(&XRayInitialized, + __sanitizer::memory_order_acquire)) { + + __sanitizer::atomic_store(&__xray::XRayPatchedFunction, + reinterpret_cast<uintptr_t>(entry), + __sanitizer::memory_order_release); + return 1; + } + return 0; +} + +int __xray_set_customevent_handler(void (*entry)(void *, size_t)) + XRAY_NEVER_INSTRUMENT { + if (__sanitizer::atomic_load(&XRayInitialized, + __sanitizer::memory_order_acquire)) { + __sanitizer::atomic_store(&__xray::XRayPatchedCustomEvent, + reinterpret_cast<uintptr_t>(entry), + __sanitizer::memory_order_release); return 1; } return 0; } + int __xray_remove_handler() XRAY_NEVER_INSTRUMENT { return __xray_set_handler(nullptr); } -std::atomic<bool> XRayPatching{false}; +int __xray_remove_customevent_handler() XRAY_NEVER_INSTRUMENT { + return __xray_set_customevent_handler(nullptr); +} + +__sanitizer::atomic_uint8_t XRayPatching{0}; using namespace __xray; @@ -115,34 +148,76 @@ public: }; template <class Function> -CleanupInvoker<Function> ScopeCleanup(Function Fn) XRAY_NEVER_INSTRUMENT { +CleanupInvoker<Function> scopeCleanup(Function Fn) XRAY_NEVER_INSTRUMENT { return CleanupInvoker<Function>{Fn}; } -// ControlPatching implements the common internals of the patching/unpatching +inline bool patchSled(const XRaySledEntry &Sled, bool Enable, + int32_t FuncId) XRAY_NEVER_INSTRUMENT { + // While we're here, we should patch the nop sled. To do that we mprotect + // the page containing the function to be writeable. + const uint64_t PageSize = GetPageSizeCached(); + void *PageAlignedAddr = + reinterpret_cast<void *>(Sled.Address & ~(PageSize - 1)); + std::size_t MProtectLen = (Sled.Address + cSledLength) - + reinterpret_cast<uint64_t>(PageAlignedAddr); + MProtectHelper Protector(PageAlignedAddr, MProtectLen); + if (Protector.MakeWriteable() == -1) { + printf("Failed mprotect: %d\n", errno); + return XRayPatchingStatus::FAILED; + } + + bool Success = false; + switch (Sled.Kind) { + case XRayEntryType::ENTRY: + Success = patchFunctionEntry(Enable, FuncId, Sled, __xray_FunctionEntry); + break; + case XRayEntryType::EXIT: + Success = patchFunctionExit(Enable, FuncId, Sled); + break; + case XRayEntryType::TAIL: + Success = patchFunctionTailExit(Enable, FuncId, Sled); + break; + case XRayEntryType::LOG_ARGS_ENTRY: + Success = patchFunctionEntry(Enable, FuncId, Sled, __xray_ArgLoggerEntry); + break; + case XRayEntryType::CUSTOM_EVENT: + Success = patchCustomEvent(Enable, FuncId, Sled); + break; + default: + Report("Unsupported sled kind '%d' @%04x\n", Sled.Address, int(Sled.Kind)); + return false; + } + return Success; +} + +// controlPatching implements the common internals of the patching/unpatching // implementation. |Enable| defines whether we're enabling or disabling the // runtime XRay instrumentation. -XRayPatchingStatus ControlPatching(bool Enable) XRAY_NEVER_INSTRUMENT { - if (!XRayInitialized.load(std::memory_order_acquire)) +XRayPatchingStatus controlPatching(bool Enable) XRAY_NEVER_INSTRUMENT { + if (!__sanitizer::atomic_load(&XRayInitialized, + __sanitizer::memory_order_acquire)) return XRayPatchingStatus::NOT_INITIALIZED; // Not initialized. - static bool NotPatching = false; - if (!XRayPatching.compare_exchange_strong(NotPatching, true, - std::memory_order_acq_rel, - std::memory_order_acquire)) { + uint8_t NotPatching = false; + if (!__sanitizer::atomic_compare_exchange_strong( + &XRayPatching, &NotPatching, true, __sanitizer::memory_order_acq_rel)) return XRayPatchingStatus::ONGOING; // Already patching. - } - bool PatchingSuccess = false; - auto XRayPatchingStatusResetter = ScopeCleanup([&PatchingSuccess] { - if (!PatchingSuccess) { - XRayPatching.store(false, std::memory_order_release); - } + uint8_t PatchingSuccess = false; + auto XRayPatchingStatusResetter = scopeCleanup([&PatchingSuccess] { + if (!PatchingSuccess) + __sanitizer::atomic_store(&XRayPatching, false, + __sanitizer::memory_order_release); }); // Step 1: Compute the function id, as a unique identifier per function in the // instrumentation map. - XRaySledMap InstrMap = XRayInstrMap.load(std::memory_order_acquire); + XRaySledMap InstrMap; + { + __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); + InstrMap = XRayInstrMap; + } if (InstrMap.Entries == 0) return XRayPatchingStatus::NOT_INITIALIZED; @@ -163,45 +238,110 @@ XRayPatchingStatus ControlPatching(bool Enable) XRAY_NEVER_INSTRUMENT { ++FuncId; CurFun = F; } - - // While we're here, we should patch the nop sled. To do that we mprotect - // the page containing the function to be writeable. - void *PageAlignedAddr = - reinterpret_cast<void *>(Sled.Address & ~(PageSize - 1)); - std::size_t MProtectLen = (Sled.Address + cSledLength) - - reinterpret_cast<uint64_t>(PageAlignedAddr); - MProtectHelper Protector(PageAlignedAddr, MProtectLen); - if (Protector.MakeWriteable() == -1) { - printf("Failed mprotect: %d\n", errno); - return XRayPatchingStatus::FAILED; - } - - bool Success = false; - switch (Sled.Kind) { - case XRayEntryType::ENTRY: - Success = patchFunctionEntry(Enable, FuncId, Sled); - break; - case XRayEntryType::EXIT: - Success = patchFunctionExit(Enable, FuncId, Sled); - break; - case XRayEntryType::TAIL: - Success = patchFunctionTailExit(Enable, FuncId, Sled); - break; - default: - Report("Unsupported sled kind: %d\n", int(Sled.Kind)); - continue; - } - (void)Success; + patchSled(Sled, Enable, FuncId); } - XRayPatching.store(false, std::memory_order_release); + __sanitizer::atomic_store(&XRayPatching, false, + __sanitizer::memory_order_release); PatchingSuccess = true; return XRayPatchingStatus::SUCCESS; } XRayPatchingStatus __xray_patch() XRAY_NEVER_INSTRUMENT { - return ControlPatching(true); + return controlPatching(true); } XRayPatchingStatus __xray_unpatch() XRAY_NEVER_INSTRUMENT { - return ControlPatching(false); + return controlPatching(false); +} + +XRayPatchingStatus patchFunction(int32_t FuncId, + bool Enable) XRAY_NEVER_INSTRUMENT { + if (!__sanitizer::atomic_load(&XRayInitialized, + __sanitizer::memory_order_acquire)) + return XRayPatchingStatus::NOT_INITIALIZED; // Not initialized. + + uint8_t NotPatching = false; + if (!__sanitizer::atomic_compare_exchange_strong( + &XRayPatching, &NotPatching, true, __sanitizer::memory_order_acq_rel)) + return XRayPatchingStatus::ONGOING; // Already patching. + + // Next, we look for the function index. + XRaySledMap InstrMap; + { + __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); + InstrMap = XRayInstrMap; + } + + // If we don't have an index, we can't patch individual functions. + if (InstrMap.Functions == 0) + return XRayPatchingStatus::NOT_INITIALIZED; + + // FuncId must be a positive number, less than the number of functions + // instrumented. + if (FuncId <= 0 || static_cast<size_t>(FuncId) > InstrMap.Functions) { + Report("Invalid function id provided: %d\n", FuncId); + return XRayPatchingStatus::FAILED; + } + + // Now we patch ths sleds for this specific function. + auto SledRange = InstrMap.SledsIndex[FuncId - 1]; + auto *f = SledRange.Begin; + auto *e = SledRange.End; + + bool SucceedOnce = false; + while (f != e) + SucceedOnce |= patchSled(*f++, Enable, FuncId); + + __sanitizer::atomic_store(&XRayPatching, false, + __sanitizer::memory_order_release); + + if (!SucceedOnce) { + Report("Failed patching any sled for function '%d'.", FuncId); + return XRayPatchingStatus::FAILED; + } + + return XRayPatchingStatus::SUCCESS; +} + +XRayPatchingStatus __xray_patch_function(int32_t FuncId) XRAY_NEVER_INSTRUMENT { + return patchFunction(FuncId, true); +} + +XRayPatchingStatus +__xray_unpatch_function(int32_t FuncId) XRAY_NEVER_INSTRUMENT { + return patchFunction(FuncId, false); +} + +int __xray_set_handler_arg1(void (*entry)(int32_t, XRayEntryType, uint64_t)) { + if (!__sanitizer::atomic_load(&XRayInitialized, + __sanitizer::memory_order_acquire)) + return 0; + + // A relaxed write might not be visible even if the current thread gets + // scheduled on a different CPU/NUMA node. We need to wait for everyone to + // have this handler installed for consistency of collected data across CPUs. + __sanitizer::atomic_store(&XRayArgLogger, reinterpret_cast<uint64_t>(entry), + __sanitizer::memory_order_release); + return 1; +} + +int __xray_remove_handler_arg1() { return __xray_set_handler_arg1(nullptr); } + +uintptr_t __xray_function_address(int32_t FuncId) XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); + if (FuncId <= 0 || static_cast<size_t>(FuncId) > XRayInstrMap.Functions) + return 0; + return XRayInstrMap.SledsIndex[FuncId - 1].Begin->Address +// On PPC, function entries are always aligned to 16 bytes. The beginning of a +// sled might be a local entry, which is always +8 based on the global entry. +// Always return the global entry. +#ifdef __PPC__ + & ~0xf +#endif + ; +} + +size_t __xray_max_function_id() XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); + return XRayInstrMap.Functions; } diff --git a/contrib/compiler-rt/lib/xray/xray_interface_internal.h b/contrib/compiler-rt/lib/xray/xray_interface_internal.h index a8434a6..4a27846 100644 --- a/contrib/compiler-rt/lib/xray/xray_interface_internal.h +++ b/contrib/compiler-rt/lib/xray/xray_interface_internal.h @@ -39,6 +39,11 @@ struct XRaySledEntry { #error "Unsupported word size." #endif }; + +struct XRayFunctionSledIndex { + const XRaySledEntry* Begin; + const XRaySledEntry* End; +}; } namespace __xray { @@ -46,15 +51,16 @@ namespace __xray { struct XRaySledMap { const XRaySledEntry *Sleds; size_t Entries; + const XRayFunctionSledIndex *SledsIndex; + size_t Functions; }; -uint64_t cycleFrequency(); - bool patchFunctionEntry(bool Enable, uint32_t FuncId, - const XRaySledEntry &Sled); + const XRaySledEntry &Sled, void (*Trampoline)()); bool patchFunctionExit(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled); bool patchFunctionTailExit(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled); +bool patchCustomEvent(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled); } // namespace __xray @@ -64,6 +70,8 @@ extern "C" { extern void __xray_FunctionEntry(); extern void __xray_FunctionExit(); extern void __xray_FunctionTailExit(); +extern void __xray_ArgLoggerEntry(); +extern void __xray_CustomEvent(); } #endif diff --git a/contrib/compiler-rt/lib/xray/xray_log_interface.cc b/contrib/compiler-rt/lib/xray/xray_log_interface.cc new file mode 100644 index 0000000..ee14ae4 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_log_interface.cc @@ -0,0 +1,69 @@ +//===-- xray_log_interface.cc ---------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a function call tracing system. +// +//===----------------------------------------------------------------------===// +#include "xray/xray_log_interface.h" + +#include "sanitizer_common/sanitizer_atomic.h" +#include "sanitizer_common/sanitizer_mutex.h" +#include "xray/xray_interface.h" +#include "xray_defs.h" + +#include <memory> + +__sanitizer::SpinMutex XRayImplMutex; +std::unique_ptr<XRayLogImpl> GlobalXRayImpl; + +void __xray_set_log_impl(XRayLogImpl Impl) XRAY_NEVER_INSTRUMENT { + if (Impl.log_init == nullptr || Impl.log_finalize == nullptr || + Impl.handle_arg0 == nullptr || Impl.flush_log == nullptr) { + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + GlobalXRayImpl.reset(); + __xray_remove_handler(); + __xray_remove_handler_arg1(); + return; + } + + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + GlobalXRayImpl.reset(new XRayLogImpl); + *GlobalXRayImpl = Impl; + __xray_set_handler(Impl.handle_arg0); +} + +void __xray_remove_log_impl() XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + GlobalXRayImpl.reset(); + __xray_remove_handler(); + __xray_remove_handler_arg1(); +} + +XRayLogInitStatus __xray_log_init(size_t BufferSize, size_t MaxBuffers, + void *Args, + size_t ArgsSize) XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + if (!GlobalXRayImpl) + return XRayLogInitStatus::XRAY_LOG_UNINITIALIZED; + return GlobalXRayImpl->log_init(BufferSize, MaxBuffers, Args, ArgsSize); +} + +XRayLogInitStatus __xray_log_finalize() XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + if (!GlobalXRayImpl) + return XRayLogInitStatus::XRAY_LOG_UNINITIALIZED; + return GlobalXRayImpl->log_finalize(); +} + +XRayLogFlushStatus __xray_log_flushLog() XRAY_NEVER_INSTRUMENT { + __sanitizer::SpinMutexLock Guard(&XRayImplMutex); + if (!GlobalXRayImpl) + return XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING; + return GlobalXRayImpl->flush_log(); +} diff --git a/contrib/compiler-rt/lib/xray/xray_mips.cc b/contrib/compiler-rt/lib/xray/xray_mips.cc new file mode 100644 index 0000000..cd86330 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_mips.cc @@ -0,0 +1,165 @@ +//===-- xray_mips.cc --------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Implementation of MIPS-specific routines (32-bit). +// +//===----------------------------------------------------------------------===// +#include "sanitizer_common/sanitizer_common.h" +#include "xray_defs.h" +#include "xray_interface_internal.h" +#include <atomic> + +namespace __xray { + +// The machine codes for some instructions used in runtime patching. +enum PatchOpcodes : uint32_t { + PO_ADDIU = 0x24000000, // addiu rt, rs, imm + PO_SW = 0xAC000000, // sw rt, offset(sp) + PO_LUI = 0x3C000000, // lui rs, %hi(address) + PO_ORI = 0x34000000, // ori rt, rs, %lo(address) + PO_JALR = 0x0000F809, // jalr rs + PO_LW = 0x8C000000, // lw rt, offset(address) + PO_B44 = 0x1000000b, // b #44 + PO_NOP = 0x0, // nop +}; + +enum RegNum : uint32_t { + RN_T0 = 0x8, + RN_T9 = 0x19, + RN_RA = 0x1F, + RN_SP = 0x1D, +}; + +inline static uint32_t encodeInstruction(uint32_t Opcode, uint32_t Rs, + uint32_t Rt, + uint32_t Imm) XRAY_NEVER_INSTRUMENT { + return (Opcode | Rs << 21 | Rt << 16 | Imm); +} + +inline static uint32_t +encodeSpecialInstruction(uint32_t Opcode, uint32_t Rs, uint32_t Rt, uint32_t Rd, + uint32_t Imm) XRAY_NEVER_INSTRUMENT { + return (Rs << 21 | Rt << 16 | Rd << 11 | Imm << 6 | Opcode); +} + +inline static bool patchSled(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled, + void (*TracingHook)()) XRAY_NEVER_INSTRUMENT { + // When |Enable| == true, + // We replace the following compile-time stub (sled): + // + // xray_sled_n: + // B .tmpN + // 11 NOPs (44 bytes) + // .tmpN + // ADDIU T9, T9, 44 + // + // With the following runtime patch: + // + // xray_sled_n (32-bit): + // addiu sp, sp, -8 ;create stack frame + // nop + // sw ra, 4(sp) ;save return address + // sw t9, 0(sp) ;save register t9 + // lui t9, %hi(__xray_FunctionEntry/Exit) + // ori t9, t9, %lo(__xray_FunctionEntry/Exit) + // lui t0, %hi(function_id) + // jalr t9 ;call Tracing hook + // ori t0, t0, %lo(function_id) ;pass function id (delay slot) + // lw t9, 0(sp) ;restore register t9 + // lw ra, 4(sp) ;restore return address + // addiu sp, sp, 8 ;delete stack frame + // + // We add 44 bytes to t9 because we want to adjust the function pointer to + // the actual start of function i.e. the address just after the noop sled. + // We do this because gp displacement relocation is emitted at the start of + // of the function i.e after the nop sled and to correctly calculate the + // global offset table address, t9 must hold the address of the instruction + // containing the gp displacement relocation. + // FIXME: Is this correct for the static relocation model? + // + // Replacement of the first 4-byte instruction should be the last and atomic + // operation, so that the user code which reaches the sled concurrently + // either jumps over the whole sled, or executes the whole sled when the + // latter is ready. + // + // When |Enable|==false, we set back the first instruction in the sled to be + // B #44 + + if (Enable) { + uint32_t LoTracingHookAddr = + reinterpret_cast<int32_t>(TracingHook) & 0xffff; + uint32_t HiTracingHookAddr = + (reinterpret_cast<int32_t>(TracingHook) >> 16) & 0xffff; + uint32_t LoFunctionID = FuncId & 0xffff; + uint32_t HiFunctionID = (FuncId >> 16) & 0xffff; + *reinterpret_cast<uint32_t *>(Sled.Address + 8) = encodeInstruction( + PatchOpcodes::PO_SW, RegNum::RN_SP, RegNum::RN_RA, 0x4); + *reinterpret_cast<uint32_t *>(Sled.Address + 12) = encodeInstruction( + PatchOpcodes::PO_SW, RegNum::RN_SP, RegNum::RN_T9, 0x0); + *reinterpret_cast<uint32_t *>(Sled.Address + 16) = encodeInstruction( + PatchOpcodes::PO_LUI, 0x0, RegNum::RN_T9, HiTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 20) = encodeInstruction( + PatchOpcodes::PO_ORI, RegNum::RN_T9, RegNum::RN_T9, LoTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 24) = encodeInstruction( + PatchOpcodes::PO_LUI, 0x0, RegNum::RN_T0, HiFunctionID); + *reinterpret_cast<uint32_t *>(Sled.Address + 28) = encodeSpecialInstruction( + PatchOpcodes::PO_JALR, RegNum::RN_T9, 0x0, RegNum::RN_RA, 0X0); + *reinterpret_cast<uint32_t *>(Sled.Address + 32) = encodeInstruction( + PatchOpcodes::PO_ORI, RegNum::RN_T0, RegNum::RN_T0, LoFunctionID); + *reinterpret_cast<uint32_t *>(Sled.Address + 36) = encodeInstruction( + PatchOpcodes::PO_LW, RegNum::RN_SP, RegNum::RN_T9, 0x0); + *reinterpret_cast<uint32_t *>(Sled.Address + 40) = encodeInstruction( + PatchOpcodes::PO_LW, RegNum::RN_SP, RegNum::RN_RA, 0x4); + *reinterpret_cast<uint32_t *>(Sled.Address + 44) = encodeInstruction( + PatchOpcodes::PO_ADDIU, RegNum::RN_SP, RegNum::RN_SP, 0x8); + uint32_t CreateStackSpaceInstr = encodeInstruction( + PatchOpcodes::PO_ADDIU, RegNum::RN_SP, RegNum::RN_SP, 0xFFF8); + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint32_t> *>(Sled.Address), + uint32_t(CreateStackSpaceInstr), std::memory_order_release); + } else { + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint32_t> *>(Sled.Address), + uint32_t(PatchOpcodes::PO_B44), std::memory_order_release); + } + return true; +} + +bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, Trampoline); +} + +bool patchFunctionExit(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); +} + +bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // FIXME: In the future we'd need to distinguish between non-tail exits and + // tail exits for better information preservation. + return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); +} + +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // FIXME: Implement in mips? + return false; +} + +} // namespace __xray + +extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT { + // FIXME: this will have to be implemented in the trampoline assembly file +} diff --git a/contrib/compiler-rt/lib/xray/xray_mips64.cc b/contrib/compiler-rt/lib/xray/xray_mips64.cc new file mode 100644 index 0000000..fa8fdd5 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_mips64.cc @@ -0,0 +1,173 @@ +//===-- xray_mips64.cc ------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Implementation of MIPS64-specific routines. +// +//===----------------------------------------------------------------------===// +#include "sanitizer_common/sanitizer_common.h" +#include "xray_defs.h" +#include "xray_interface_internal.h" +#include <atomic> + +namespace __xray { + +// The machine codes for some instructions used in runtime patching. +enum PatchOpcodes : uint32_t { + PO_DADDIU = 0x64000000, // daddiu rt, rs, imm + PO_SD = 0xFC000000, // sd rt, base(offset) + PO_LUI = 0x3C000000, // lui rt, imm + PO_ORI = 0x34000000, // ori rt, rs, imm + PO_DSLL = 0x00000038, // dsll rd, rt, sa + PO_JALR = 0x00000009, // jalr rs + PO_LD = 0xDC000000, // ld rt, base(offset) + PO_B60 = 0x1000000f, // b #60 + PO_NOP = 0x0, // nop +}; + +enum RegNum : uint32_t { + RN_T0 = 0xC, + RN_T9 = 0x19, + RN_RA = 0x1F, + RN_SP = 0x1D, +}; + +inline static uint32_t encodeInstruction(uint32_t Opcode, uint32_t Rs, + uint32_t Rt, + uint32_t Imm) XRAY_NEVER_INSTRUMENT { + return (Opcode | Rs << 21 | Rt << 16 | Imm); +} + +inline static uint32_t +encodeSpecialInstruction(uint32_t Opcode, uint32_t Rs, uint32_t Rt, uint32_t Rd, + uint32_t Imm) XRAY_NEVER_INSTRUMENT { + return (Rs << 21 | Rt << 16 | Rd << 11 | Imm << 6 | Opcode); +} + +inline static bool patchSled(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled, + void (*TracingHook)()) XRAY_NEVER_INSTRUMENT { + // When |Enable| == true, + // We replace the following compile-time stub (sled): + // + // xray_sled_n: + // B .tmpN + // 15 NOPs (60 bytes) + // .tmpN + // + // With the following runtime patch: + // + // xray_sled_n (64-bit): + // daddiu sp, sp, -16 ;create stack frame + // nop + // sd ra, 8(sp) ;save return address + // sd t9, 0(sp) ;save register t9 + // lui t9, %highest(__xray_FunctionEntry/Exit) + // ori t9, t9, %higher(__xray_FunctionEntry/Exit) + // dsll t9, t9, 16 + // ori t9, t9, %hi(__xray_FunctionEntry/Exit) + // dsll t9, t9, 16 + // ori t9, t9, %lo(__xray_FunctionEntry/Exit) + // lui t0, %hi(function_id) + // jalr t9 ;call Tracing hook + // ori t0, t0, %lo(function_id) ;pass function id (delay slot) + // ld t9, 0(sp) ;restore register t9 + // ld ra, 8(sp) ;restore return address + // daddiu sp, sp, 16 ;delete stack frame + // + // Replacement of the first 4-byte instruction should be the last and atomic + // operation, so that the user code which reaches the sled concurrently + // either jumps over the whole sled, or executes the whole sled when the + // latter is ready. + // + // When |Enable|==false, we set back the first instruction in the sled to be + // B #60 + + if (Enable) { + uint32_t LoTracingHookAddr = + reinterpret_cast<int64_t>(TracingHook) & 0xffff; + uint32_t HiTracingHookAddr = + (reinterpret_cast<int64_t>(TracingHook) >> 16) & 0xffff; + uint32_t HigherTracingHookAddr = + (reinterpret_cast<int64_t>(TracingHook) >> 32) & 0xffff; + uint32_t HighestTracingHookAddr = + (reinterpret_cast<int64_t>(TracingHook) >> 48) & 0xffff; + uint32_t LoFunctionID = FuncId & 0xffff; + uint32_t HiFunctionID = (FuncId >> 16) & 0xffff; + *reinterpret_cast<uint32_t *>(Sled.Address + 8) = encodeInstruction( + PatchOpcodes::PO_SD, RegNum::RN_SP, RegNum::RN_RA, 0x8); + *reinterpret_cast<uint32_t *>(Sled.Address + 12) = encodeInstruction( + PatchOpcodes::PO_SD, RegNum::RN_SP, RegNum::RN_T9, 0x0); + *reinterpret_cast<uint32_t *>(Sled.Address + 16) = encodeInstruction( + PatchOpcodes::PO_LUI, 0x0, RegNum::RN_T9, HighestTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 20) = + encodeInstruction(PatchOpcodes::PO_ORI, RegNum::RN_T9, RegNum::RN_T9, + HigherTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 24) = encodeSpecialInstruction( + PatchOpcodes::PO_DSLL, 0x0, RegNum::RN_T9, RegNum::RN_T9, 0x10); + *reinterpret_cast<uint32_t *>(Sled.Address + 28) = encodeInstruction( + PatchOpcodes::PO_ORI, RegNum::RN_T9, RegNum::RN_T9, HiTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 32) = encodeSpecialInstruction( + PatchOpcodes::PO_DSLL, 0x0, RegNum::RN_T9, RegNum::RN_T9, 0x10); + *reinterpret_cast<uint32_t *>(Sled.Address + 36) = encodeInstruction( + PatchOpcodes::PO_ORI, RegNum::RN_T9, RegNum::RN_T9, LoTracingHookAddr); + *reinterpret_cast<uint32_t *>(Sled.Address + 40) = encodeInstruction( + PatchOpcodes::PO_LUI, 0x0, RegNum::RN_T0, HiFunctionID); + *reinterpret_cast<uint32_t *>(Sled.Address + 44) = encodeSpecialInstruction( + PatchOpcodes::PO_JALR, RegNum::RN_T9, 0x0, RegNum::RN_RA, 0X0); + *reinterpret_cast<uint32_t *>(Sled.Address + 48) = encodeInstruction( + PatchOpcodes::PO_ORI, RegNum::RN_T0, RegNum::RN_T0, LoFunctionID); + *reinterpret_cast<uint32_t *>(Sled.Address + 52) = encodeInstruction( + PatchOpcodes::PO_LD, RegNum::RN_SP, RegNum::RN_T9, 0x0); + *reinterpret_cast<uint32_t *>(Sled.Address + 56) = encodeInstruction( + PatchOpcodes::PO_LD, RegNum::RN_SP, RegNum::RN_RA, 0x8); + *reinterpret_cast<uint32_t *>(Sled.Address + 60) = encodeInstruction( + PatchOpcodes::PO_DADDIU, RegNum::RN_SP, RegNum::RN_SP, 0x10); + uint32_t CreateStackSpace = encodeInstruction( + PatchOpcodes::PO_DADDIU, RegNum::RN_SP, RegNum::RN_SP, 0xfff0); + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint32_t> *>(Sled.Address), + CreateStackSpace, std::memory_order_release); + } else { + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint32_t> *>(Sled.Address), + uint32_t(PatchOpcodes::PO_B60), std::memory_order_release); + } + return true; +} + +bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, Trampoline); +} + +bool patchFunctionExit(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); +} + +bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // FIXME: In the future we'd need to distinguish between non-tail exits and + // tail exits for better information preservation. + return patchSled(Enable, FuncId, Sled, __xray_FunctionExit); +} + +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // FIXME: Implement in mips64? + return false; +} +} // namespace __xray + +extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT { + // FIXME: this will have to be implemented in the trampoline assembly file +} diff --git a/contrib/compiler-rt/lib/xray/xray_never_instrument.txt b/contrib/compiler-rt/lib/xray/xray_never_instrument.txt new file mode 100644 index 0000000..7fa48dd --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_never_instrument.txt @@ -0,0 +1,6 @@ +# List of function matchers common to C/C++ applications that make sense to +# never instrument. You can use this as an argument to +# -fxray-never-instrument=<path> along with your project-specific lists. + +# Never instrument any function whose symbol starts with __xray. +fun:__xray* diff --git a/contrib/compiler-rt/lib/xray/xray_powerpc64.cc b/contrib/compiler-rt/lib/xray/xray_powerpc64.cc new file mode 100644 index 0000000..ab03cb1 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_powerpc64.cc @@ -0,0 +1,106 @@ +//===-- xray_powerpc64.cc ---------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Implementation of powerpc64 and powerpc64le routines. +// +//===----------------------------------------------------------------------===// +#include "sanitizer_common/sanitizer_common.h" +#include "xray_defs.h" +#include "xray_interface_internal.h" +#include "xray_utils.h" +#include <atomic> +#include <cassert> +#include <cstring> + +#ifndef __LITTLE_ENDIAN__ +#error powerpc64 big endian is not supported for now. +#endif + +namespace { + +constexpr unsigned long long JumpOverInstNum = 7; + +void clearCache(void *Addr, size_t Len) { + const size_t LineSize = 32; + + const intptr_t Mask = ~(LineSize - 1); + const intptr_t StartLine = ((intptr_t)Addr) & Mask; + const intptr_t EndLine = ((intptr_t)Addr + Len + LineSize - 1) & Mask; + + for (intptr_t Line = StartLine; Line < EndLine; Line += LineSize) + asm volatile("dcbf 0, %0" : : "r"(Line)); + asm volatile("sync"); + + for (intptr_t Line = StartLine; Line < EndLine; Line += LineSize) + asm volatile("icbi 0, %0" : : "r"(Line)); + asm volatile("isync"); +} + +} // namespace + +extern "C" void __clear_cache(void *start, void *end); + +namespace __xray { + +bool patchFunctionEntry(const bool Enable, uint32_t FuncId, + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { + if (Enable) { + // lis 0, FuncId[16..32] + // li 0, FuncId[0..15] + *reinterpret_cast<uint64_t *>(Sled.Address) = + (0x3c000000ull + (FuncId >> 16)) + + ((0x60000000ull + (FuncId & 0xffff)) << 32); + } else { + // b +JumpOverInstNum instructions. + *reinterpret_cast<uint32_t *>(Sled.Address) = + 0x48000000ull + (JumpOverInstNum << 2); + } + clearCache(reinterpret_cast<void *>(Sled.Address), 8); + return true; +} + +bool patchFunctionExit(const bool Enable, uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + if (Enable) { + // lis 0, FuncId[16..32] + // li 0, FuncId[0..15] + *reinterpret_cast<uint64_t *>(Sled.Address) = + (0x3c000000ull + (FuncId >> 16)) + + ((0x60000000ull + (FuncId & 0xffff)) << 32); + } else { + // Copy the blr/b instruction after JumpOverInstNum instructions. + *reinterpret_cast<uint32_t *>(Sled.Address) = + *(reinterpret_cast<uint32_t *>(Sled.Address) + JumpOverInstNum); + } + clearCache(reinterpret_cast<void *>(Sled.Address), 8); + return true; +} + +bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + return patchFunctionExit(Enable, FuncId, Sled); +} + +// FIXME: Maybe implement this better? +bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; } + +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // FIXME: Implement in powerpc64? + return false; +} + +} // namespace __xray + +extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT { + // FIXME: this will have to be implemented in the trampoline assembly file +} diff --git a/contrib/compiler-rt/lib/xray/xray_powerpc64.inc b/contrib/compiler-rt/lib/xray/xray_powerpc64.inc new file mode 100644 index 0000000..c1a1bac --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_powerpc64.inc @@ -0,0 +1,37 @@ +//===-- xray_powerpc64.inc --------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +//===----------------------------------------------------------------------===// + +#include <cstdint> +#include <mutex> +#include <sys/platform/ppc.h> + +#include "xray_defs.h" + +namespace __xray { + +ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT { + CPU = 0; + return __ppc_get_timebase(); +} + +inline uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT { + static std::mutex M; + std::lock_guard<std::mutex> Guard(M); + return __ppc_get_timebase_freq(); +} + +inline bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { + return true; +} + +} // namespace __xray diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_AArch64.S b/contrib/compiler-rt/lib/xray/xray_trampoline_AArch64.S index f1a471c..4d1b04f 100644 --- a/contrib/compiler-rt/lib/xray/xray_trampoline_AArch64.S +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_AArch64.S @@ -1,3 +1,5 @@ +#include "../builtins/assembly.h" + .text /* The variable containing the handler function pointer */ .global _ZN6__xray19XRayPatchedFunctionE @@ -87,3 +89,56 @@ FunctionExit_restore: LDP X3, X4, [SP], #16 LDP X1, X2, [SP], #16 RET + + /* Word-aligned function entry point */ + .p2align 2 + /* Let C/C++ see the symbol */ + .global __xray_FunctionTailExit + .type __xray_FunctionTailExit, %function + /* In C++ it is void extern "C" __xray_FunctionTailExit(uint32_t FuncId) + with FuncId passed in W0 register. */ +__xray_FunctionTailExit: + /* Move the return address beyond the end of sled data. The 12 bytes of + data are inserted in the code of the runtime patch, between the call + instruction and the instruction returned into. The data contains 32 + bits of instrumented function ID and 64 bits of the address of + the current trampoline. */ + ADD X30, X30, #12 + /* Push the registers which may be modified by the handler function */ + STP X1, X2, [SP, #-16]! + STP X3, X4, [SP, #-16]! + STP X5, X6, [SP, #-16]! + STP X7, X30, [SP, #-16]! + /* Push the parameters of the tail called function */ + STP Q0, Q1, [SP, #-32]! + STP Q2, Q3, [SP, #-32]! + STP Q4, Q5, [SP, #-32]! + STP Q6, Q7, [SP, #-32]! + /* Load the address of _ZN6__xray19XRayPatchedFunctionE into X1 */ + LDR X1, =_ZN6__xray19XRayPatchedFunctionE + /* Load the handler function pointer into X2 */ + LDR X2, [X1] + /* Handler address is nullptr if handler is not set */ + CMP X2, #0 + BEQ FunctionTailExit_restore + /* Function ID is already in W0 (the first parameter). + X1=2 means that we are tracing a tail exit event, but before the + logging part of XRay is ready, we pretend that here a normal function + exit happens, so we give the handler code 1 */ + MOV X1, #1 + /* Call the handler with 2 parameters in W0 and X1 */ + BLR X2 +FunctionTailExit_restore: + /* Pop the parameters of the tail called function */ + LDP Q6, Q7, [SP], #32 + LDP Q4, Q5, [SP], #32 + LDP Q2, Q3, [SP], #32 + LDP Q0, Q1, [SP], #32 + /* Pop the registers which may be modified by the handler function */ + LDP X7, X30, [SP], #16 + LDP X5, X6, [SP], #16 + LDP X3, X4, [SP], #16 + LDP X1, X2, [SP], #16 + RET + +NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_arm.S b/contrib/compiler-rt/lib/xray/xray_trampoline_arm.S index 5d87c97..71dbee6 100644 --- a/contrib/compiler-rt/lib/xray/xray_trampoline_arm.S +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_arm.S @@ -1,8 +1,11 @@ +#include "../builtins/assembly.h" + .syntax unified .arch armv6t2 .fpu vfpv2 .code 32 .global _ZN6__xray19XRayPatchedFunctionE + @ Word-aligned function entry point .p2align 2 @ Let C/C++ see the symbol @@ -63,3 +66,37 @@ FunctionExit_restore: @ Restore the floating-point return value of the instrumented function VPOP {d0} POP {r1-r3,pc} + + @ Word-aligned function entry point + .p2align 2 + @ Let C/C++ see the symbol + .global __xray_FunctionTailExit + @ It preserves all registers except r0, r12(ip), r14(lr) and r15(pc) + @ Assume that "q" part of the floating-point registers is not used + @ for passing parameters to C/C++ functions. + .type __xray_FunctionTailExit, %function + @ In C++ it is void extern "C" __xray_FunctionTailExit(uint32_t FuncId) + @ with FuncId passed in r0 register. +__xray_FunctionTailExit: + PUSH {r1-r3,lr} + @ Save floating-point parameters of the instrumented function + VPUSH {d0-d7} + MOVW r1,#:lower16:_ZN6__xray19XRayPatchedFunctionE + MOVT r1,#:upper16:_ZN6__xray19XRayPatchedFunctionE + LDR r2, [r1] + @ Handler address is nullptr if handler is not set + CMP r2, #0 + BEQ FunctionTailExit_restore + @ Function ID is already in r0 (the first parameter). + @ r1=2 means that we are tracing a tail exit event + @ But before the logging part of XRay is ready, we pretend that here a + @ normal function exit happens, so we give the handler code 1 + MOV r1, #1 + @ Call the handler with 2 parameters in r0 and r1 + BLX r2 +FunctionTailExit_restore: + @ Restore floating-point parameters of the instrumented function + VPOP {d0-d7} + POP {r1-r3,pc} + +NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_mips.S b/contrib/compiler-rt/lib/xray/xray_trampoline_mips.S new file mode 100644 index 0000000..39a1a3a --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_mips.S @@ -0,0 +1,110 @@ +//===-- xray_trampoline_mips.s ----------------------------------*- ASM -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// This implements the MIPS-specific assembler for the trampolines. +// +//===----------------------------------------------------------------------===// + + .text + .file "xray_trampoline_mips.S" + .globl __xray_FunctionEntry + .p2align 2 + .type __xray_FunctionEntry,@function +__xray_FunctionEntry: + .cfi_startproc + .set noreorder + .cpload $t9 + .set reorder + // Save argument registers before doing any actual work + .cfi_def_cfa_offset 36 + addiu $sp, $sp, -36 + sw $ra, 32($sp) + .cfi_offset 31, -4 + sw $a3, 28($sp) + sw $a2, 24($sp) + sw $a1, 20($sp) + sw $a0, 16($sp) + sdc1 $f14, 8($sp) + sdc1 $f12, 0($sp) + + la $t9, _ZN6__xray19XRayPatchedFunctionE + lw $t9, 0($t9) + + beqz $t9, FunctionEntry_restore + + // a1=0 means that we are tracing an entry event + move $a1, $zero + // Function ID is in t0 (the first parameter). + move $a0, $t0 + jalr $t9 + +FunctionEntry_restore: + // Restore argument registers + ldc1 $f12, 0($sp) + ldc1 $f14, 8($sp) + lw $a0, 16($sp) + lw $a1, 20($sp) + lw $a2, 24($sp) + lw $a3, 28($sp) + lw $ra, 32($sp) + addiu $sp, $sp, 36 + jr $ra +FunctionEntry_end: + .size __xray_FunctionEntry, FunctionEntry_end-__xray_FunctionEntry + .cfi_endproc + + .text + .globl __xray_FunctionExit + .p2align 2 + .type __xray_FunctionExit,@function +__xray_FunctionExit: + .cfi_startproc + .set noreorder + .cpload $t9 + .set reorder + // Save return registers before doing any actual work. + .cfi_def_cfa_offset 36 + addiu $sp, $sp, -36 + sw $ra, 32($sp) + .cfi_offset 31, -4 + sw $a1, 28($sp) + sw $a0, 24($sp) + sw $v1, 20($sp) + sw $v0, 16($sp) + sdc1 $f2, 8($sp) + sdc1 $f0, 0($sp) + + la $t9, _ZN6__xray19XRayPatchedFunctionE + lw $t9, 0($t9) + + beqz $t9, FunctionExit_restore + + // a1=1 means that we are tracing an exit event + li $a1, 1 + // Function ID is in t0 (the first parameter). + move $a0, $t0 + jalr $t9 + +FunctionExit_restore: + // Restore return registers + ldc1 $f0, 0($sp) + ldc1 $f2, 8($sp) + lw $v0, 16($sp) + lw $v1, 20($sp) + lw $a0, 24($sp) + lw $a1, 28($sp) + lw $ra, 32($sp) + addiu $sp, $sp, 36 + jr $ra + +FunctionExit_end: + .size __xray_FunctionExit, FunctionExit_end-__xray_FunctionExit + .cfi_endproc diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_mips64.S b/contrib/compiler-rt/lib/xray/xray_trampoline_mips64.S new file mode 100644 index 0000000..9cbc7e1 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_mips64.S @@ -0,0 +1,136 @@ +//===-- xray_trampoline_mips64.s --------------------------------*- ASM -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// This implements the MIPS64-specific assembler for the trampolines. +// +//===----------------------------------------------------------------------===// + + .text + .file "xray_trampoline_mips64.S" + .globl __xray_FunctionEntry + .p2align 2 + .type __xray_FunctionEntry,@function +__xray_FunctionEntry: + .cfi_startproc + // Save argument registers before doing any actual work. + .cfi_def_cfa_offset 144 + daddiu $sp, $sp, -144 + sd $ra, 136($sp) + .cfi_offset 31, -8 + sd $gp, 128($sp) + sd $a7, 120($sp) + sd $a6, 112($sp) + sd $a5, 104($sp) + sd $a4, 96($sp) + sd $a3, 88($sp) + sd $a2, 80($sp) + sd $a1, 72($sp) + sd $a0, 64($sp) + sdc1 $f19, 56($sp) + sdc1 $f18, 48($sp) + sdc1 $f17, 40($sp) + sdc1 $f16, 32($sp) + sdc1 $f15, 24($sp) + sdc1 $f14, 16($sp) + sdc1 $f13, 8($sp) + sdc1 $f12, 0($sp) + + lui $gp, %hi(%neg(%gp_rel(__xray_FunctionEntry))) + daddu $gp, $gp, $t9 + daddiu $gp ,$gp, %lo(%neg(%gp_rel(__xray_FunctionEntry))) + + dla $t9, _ZN6__xray19XRayPatchedFunctionE + ld $t9, 0($t9) + + beqz $t9, FunctionEntry_restore + + // a1=0 means that we are tracing an entry event + move $a1, $zero + // Function ID is in t0 (the first parameter). + move $a0, $t0 + jalr $t9 + +FunctionEntry_restore: + // Restore argument registers + ldc1 $f12, 0($sp) + ldc1 $f13, 8($sp) + ldc1 $f14, 16($sp) + ldc1 $f15, 24($sp) + ldc1 $f16, 32($sp) + ldc1 $f17, 40($sp) + ldc1 $f18, 48($sp) + ldc1 $f19, 56($sp) + ld $a0, 64($sp) + ld $a1, 72($sp) + ld $a2, 80($sp) + ld $a3, 88($sp) + ld $a4, 96($sp) + ld $a5, 104($sp) + ld $a6, 112($sp) + ld $a7, 120($sp) + ld $gp, 128($sp) + ld $ra, 136($sp) + daddiu $sp, $sp, 144 + jr $ra +FunctionEntry_end: + .size __xray_FunctionEntry, FunctionEntry_end-__xray_FunctionEntry + .cfi_endproc + + .text + .globl __xray_FunctionExit + .p2align 2 + .type __xray_FunctionExit,@function +__xray_FunctionExit: + .cfi_startproc + // Save return registers before doing any actual work. + .cfi_def_cfa_offset 64 + daddiu $sp, $sp, -64 + sd $ra, 56($sp) + .cfi_offset 31, -8 + sd $gp, 48($sp) + sd $a0, 40($sp) + sd $v1, 32($sp) + sd $v0, 24($sp) + sdc1 $f2, 16($sp) + sdc1 $f1, 8($sp) + sdc1 $f0, 0($sp) + + lui $gp, %hi(%neg(%gp_rel(__xray_FunctionExit))) + daddu $gp, $gp, $t9 + daddiu $gp ,$gp, %lo(%neg(%gp_rel(__xray_FunctionExit))) + + dla $t9, _ZN6__xray19XRayPatchedFunctionE + ld $t9, 0($t9) + + beqz $t9, FunctionExit_restore + + // a1=1 means that we are tracing an exit event + li $a1, 1 + // Function ID is in t0 (the first parameter). + move $a0, $t0 + jalr $t9 + +FunctionExit_restore: + // Restore return registers + ldc1 $f0, 0($sp) + ldc1 $f1, 8($sp) + ldc1 $f2, 16($sp) + ld $v0, 24($sp) + ld $v1, 32($sp) + ld $a0, 40($sp) + ld $gp, 48($sp) + ld $ra, 56($sp) + daddiu $sp, $sp, 64 + jr $ra + +FunctionExit_end: + .size __xray_FunctionExit, FunctionExit_end-__xray_FunctionExit + .cfi_endproc diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64.cc b/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64.cc new file mode 100644 index 0000000..878c469 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64.cc @@ -0,0 +1,15 @@ +#include <atomic> +#include <xray/xray_interface.h> + +namespace __xray { + +extern std::atomic<void (*)(int32_t, XRayEntryType)> XRayPatchedFunction; + +// Implement this in C++ instead of assembly, to avoid dealing with ToC by hand. +void CallXRayPatchedFunction(int32_t FuncId, XRayEntryType Type) { + auto fptr = __xray::XRayPatchedFunction.load(); + if (fptr != nullptr) + (*fptr)(FuncId, Type); +} + +} // namespace __xray diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64_asm.S b/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64_asm.S new file mode 100644 index 0000000..250e2e5b --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_powerpc64_asm.S @@ -0,0 +1,235 @@ + .text + .abiversion 2 + .globl __xray_FunctionEntry + .p2align 4 +__xray_FunctionEntry: + std 0, 16(1) + stdu 1, -408(1) +# Spill r3-r10, f1-f13, and vsr34-vsr45, which are parameter registers. +# If this appears to be slow, the caller needs to pass in number of generic, +# floating point, and vector parameters, so that we only spill those live ones. + std 3, 32(1) + ld 3, 400(1) # FuncId + std 4, 40(1) + std 5, 48(1) + std 6, 56(1) + std 7, 64(1) + std 8, 72(1) + std 9, 80(1) + std 10, 88(1) + addi 4, 1, 96 + stxsdx 1, 0, 4 + addi 4, 1, 104 + stxsdx 2, 0, 4 + addi 4, 1, 112 + stxsdx 3, 0, 4 + addi 4, 1, 120 + stxsdx 4, 0, 4 + addi 4, 1, 128 + stxsdx 5, 0, 4 + addi 4, 1, 136 + stxsdx 6, 0, 4 + addi 4, 1, 144 + stxsdx 7, 0, 4 + addi 4, 1, 152 + stxsdx 8, 0, 4 + addi 4, 1, 160 + stxsdx 9, 0, 4 + addi 4, 1, 168 + stxsdx 10, 0, 4 + addi 4, 1, 176 + stxsdx 11, 0, 4 + addi 4, 1, 184 + stxsdx 12, 0, 4 + addi 4, 1, 192 + stxsdx 13, 0, 4 + addi 4, 1, 200 + stxvd2x 34, 0, 4 + addi 4, 1, 216 + stxvd2x 35, 0, 4 + addi 4, 1, 232 + stxvd2x 36, 0, 4 + addi 4, 1, 248 + stxvd2x 37, 0, 4 + addi 4, 1, 264 + stxvd2x 38, 0, 4 + addi 4, 1, 280 + stxvd2x 39, 0, 4 + addi 4, 1, 296 + stxvd2x 40, 0, 4 + addi 4, 1, 312 + stxvd2x 41, 0, 4 + addi 4, 1, 328 + stxvd2x 42, 0, 4 + addi 4, 1, 344 + stxvd2x 43, 0, 4 + addi 4, 1, 360 + stxvd2x 44, 0, 4 + addi 4, 1, 376 + stxvd2x 45, 0, 4 + std 2, 392(1) + mflr 0 + std 0, 400(1) + + li 4, 0 + bl _ZN6__xray23CallXRayPatchedFunctionEi13XRayEntryType + nop + + addi 4, 1, 96 + lxsdx 1, 0, 4 + addi 4, 1, 104 + lxsdx 2, 0, 4 + addi 4, 1, 112 + lxsdx 3, 0, 4 + addi 4, 1, 120 + lxsdx 4, 0, 4 + addi 4, 1, 128 + lxsdx 5, 0, 4 + addi 4, 1, 136 + lxsdx 6, 0, 4 + addi 4, 1, 144 + lxsdx 7, 0, 4 + addi 4, 1, 152 + lxsdx 8, 0, 4 + addi 4, 1, 160 + lxsdx 9, 0, 4 + addi 4, 1, 168 + lxsdx 10, 0, 4 + addi 4, 1, 176 + lxsdx 11, 0, 4 + addi 4, 1, 184 + lxsdx 12, 0, 4 + addi 4, 1, 192 + lxsdx 13, 0, 4 + addi 4, 1, 200 + lxvd2x 34, 0, 4 + addi 4, 1, 216 + lxvd2x 35, 0, 4 + addi 4, 1, 232 + lxvd2x 36, 0, 4 + addi 4, 1, 248 + lxvd2x 37, 0, 4 + addi 4, 1, 264 + lxvd2x 38, 0, 4 + addi 4, 1, 280 + lxvd2x 39, 0, 4 + addi 4, 1, 296 + lxvd2x 40, 0, 4 + addi 4, 1, 312 + lxvd2x 41, 0, 4 + addi 4, 1, 328 + lxvd2x 42, 0, 4 + addi 4, 1, 344 + lxvd2x 43, 0, 4 + addi 4, 1, 360 + lxvd2x 44, 0, 4 + addi 4, 1, 376 + lxvd2x 45, 0, 4 + ld 0, 400(1) + mtlr 0 + ld 2, 392(1) + ld 3, 32(1) + ld 4, 40(1) + ld 5, 48(1) + ld 6, 56(1) + ld 7, 64(1) + ld 8, 72(1) + ld 9, 80(1) + ld 10, 88(1) + + addi 1, 1, 408 + ld 0, 16(1) + blr + + .globl __xray_FunctionExit + .p2align 4 +__xray_FunctionExit: + std 0, 16(1) + stdu 1, -256(1) +# Spill r3-r4, f1-f8, and vsr34-vsr41, which are return registers. +# If this appears to be slow, the caller needs to pass in number of generic, +# floating point, and vector parameters, so that we only spill those live ones. + std 3, 32(1) + ld 3, 248(1) # FuncId + std 4, 40(1) + addi 4, 1, 48 + stxsdx 1, 0, 4 + addi 4, 1, 56 + stxsdx 2, 0, 4 + addi 4, 1, 64 + stxsdx 3, 0, 4 + addi 4, 1, 72 + stxsdx 4, 0, 4 + addi 4, 1, 80 + stxsdx 5, 0, 4 + addi 4, 1, 88 + stxsdx 6, 0, 4 + addi 4, 1, 96 + stxsdx 7, 0, 4 + addi 4, 1, 104 + stxsdx 8, 0, 4 + addi 4, 1, 112 + stxvd2x 34, 0, 4 + addi 4, 1, 128 + stxvd2x 35, 0, 4 + addi 4, 1, 144 + stxvd2x 36, 0, 4 + addi 4, 1, 160 + stxvd2x 37, 0, 4 + addi 4, 1, 176 + stxvd2x 38, 0, 4 + addi 4, 1, 192 + stxvd2x 39, 0, 4 + addi 4, 1, 208 + stxvd2x 40, 0, 4 + addi 4, 1, 224 + stxvd2x 41, 0, 4 + std 2, 240(1) + mflr 0 + std 0, 248(1) + + li 4, 1 + bl _ZN6__xray23CallXRayPatchedFunctionEi13XRayEntryType + nop + + addi 4, 1, 48 + lxsdx 1, 0, 4 + addi 4, 1, 56 + lxsdx 2, 0, 4 + addi 4, 1, 64 + lxsdx 3, 0, 4 + addi 4, 1, 72 + lxsdx 4, 0, 4 + addi 4, 1, 80 + lxsdx 5, 0, 4 + addi 4, 1, 88 + lxsdx 6, 0, 4 + addi 4, 1, 96 + lxsdx 7, 0, 4 + addi 4, 1, 104 + lxsdx 8, 0, 4 + addi 4, 1, 112 + lxvd2x 34, 0, 4 + addi 4, 1, 128 + lxvd2x 35, 0, 4 + addi 4, 1, 144 + lxvd2x 36, 0, 4 + addi 4, 1, 160 + lxvd2x 37, 0, 4 + addi 4, 1, 176 + lxvd2x 38, 0, 4 + addi 4, 1, 192 + lxvd2x 39, 0, 4 + addi 4, 1, 208 + lxvd2x 40, 0, 4 + addi 4, 1, 224 + lxvd2x 41, 0, 4 + ld 0, 248(1) + mtlr 0 + ld 2, 240(1) + ld 3, 32(1) + ld 4, 40(1) + + addi 1, 1, 256 + ld 0, 16(1) + blr diff --git a/contrib/compiler-rt/lib/xray/xray_trampoline_x86_64.S b/contrib/compiler-rt/lib/xray/xray_trampoline_x86_64.S index d90c30c..b59eedc 100644 --- a/contrib/compiler-rt/lib/xray/xray_trampoline_x86_64.S +++ b/contrib/compiler-rt/lib/xray/xray_trampoline_x86_64.S @@ -13,54 +13,64 @@ // //===----------------------------------------------------------------------===// +#include "../builtins/assembly.h" + .macro SAVE_REGISTERS - subq $200, %rsp - movupd %xmm0, 184(%rsp) - movupd %xmm1, 168(%rsp) - movupd %xmm2, 152(%rsp) - movupd %xmm3, 136(%rsp) - movupd %xmm4, 120(%rsp) - movupd %xmm5, 104(%rsp) - movupd %xmm6, 88(%rsp) - movupd %xmm7, 72(%rsp) - movq %rdi, 64(%rsp) - movq %rax, 56(%rsp) - movq %rdx, 48(%rsp) - movq %rsi, 40(%rsp) - movq %rcx, 32(%rsp) - movq %r8, 24(%rsp) - movq %r9, 16(%rsp) + subq $192, %rsp + .cfi_def_cfa_offset 200 + // At this point, the stack pointer should be aligned to an 8-byte boundary, + // because any call instructions that come after this will add another 8 + // bytes and therefore align it to 16-bytes. + movq %rbp, 184(%rsp) + movupd %xmm0, 168(%rsp) + movupd %xmm1, 152(%rsp) + movupd %xmm2, 136(%rsp) + movupd %xmm3, 120(%rsp) + movupd %xmm4, 104(%rsp) + movupd %xmm5, 88(%rsp) + movupd %xmm6, 72(%rsp) + movupd %xmm7, 56(%rsp) + movq %rdi, 48(%rsp) + movq %rax, 40(%rsp) + movq %rdx, 32(%rsp) + movq %rsi, 24(%rsp) + movq %rcx, 16(%rsp) + movq %r8, 8(%rsp) + movq %r9, 0(%rsp) .endm .macro RESTORE_REGISTERS - movupd 184(%rsp), %xmm0 - movupd 168(%rsp), %xmm1 - movupd 152(%rsp), %xmm2 - movupd 136(%rsp), %xmm3 - movupd 120(%rsp), %xmm4 - movupd 104(%rsp), %xmm5 - movupd 88(%rsp) , %xmm6 - movupd 72(%rsp) , %xmm7 - movq 64(%rsp), %rdi - movq 56(%rsp), %rax - movq 48(%rsp), %rdx - movq 40(%rsp), %rsi - movq 32(%rsp), %rcx - movq 24(%rsp), %r8 - movq 16(%rsp), %r9 - addq $200, %rsp + movq 184(%rsp), %rbp + movupd 168(%rsp), %xmm0 + movupd 152(%rsp), %xmm1 + movupd 136(%rsp), %xmm2 + movupd 120(%rsp), %xmm3 + movupd 104(%rsp), %xmm4 + movupd 88(%rsp), %xmm5 + movupd 72(%rsp) , %xmm6 + movupd 56(%rsp) , %xmm7 + movq 48(%rsp), %rdi + movq 40(%rsp), %rax + movq 32(%rsp), %rdx + movq 24(%rsp), %rsi + movq 16(%rsp), %rcx + movq 8(%rsp), %r8 + movq 0(%rsp), %r9 + addq $192, %rsp + .cfi_def_cfa_offset 8 .endm .text .file "xray_trampoline_x86.S" + +//===----------------------------------------------------------------------===// + .globl __xray_FunctionEntry .align 16, 0x90 .type __xray_FunctionEntry,@function __xray_FunctionEntry: .cfi_startproc - pushq %rbp - .cfi_def_cfa_offset 16 SAVE_REGISTERS // This load has to be atomic, it's concurrent with __xray_patch(). @@ -75,12 +85,13 @@ __xray_FunctionEntry: callq *%rax .Ltmp0: RESTORE_REGISTERS - popq %rbp retq .Ltmp1: .size __xray_FunctionEntry, .Ltmp1-__xray_FunctionEntry .cfi_endproc +//===----------------------------------------------------------------------===// + .globl __xray_FunctionExit .align 16, 0x90 .type __xray_FunctionExit,@function @@ -89,14 +100,13 @@ __xray_FunctionExit: // Save the important registers first. Since we're assuming that this // function is only jumped into, we only preserve the registers for // returning. - pushq %rbp - .cfi_def_cfa_offset 16 subq $56, %rsp - .cfi_def_cfa_offset 32 - movupd %xmm0, 40(%rsp) - movupd %xmm1, 24(%rsp) - movq %rax, 16(%rsp) - movq %rdx, 8(%rsp) + .cfi_def_cfa_offset 64 + movq %rbp, 48(%rsp) + movupd %xmm0, 32(%rsp) + movupd %xmm1, 16(%rsp) + movq %rax, 8(%rsp) + movq %rdx, 0(%rsp) movq _ZN6__xray19XRayPatchedFunctionE(%rip), %rax testq %rax,%rax je .Ltmp2 @@ -106,17 +116,20 @@ __xray_FunctionExit: callq *%rax .Ltmp2: // Restore the important registers. - movupd 40(%rsp), %xmm0 - movupd 24(%rsp), %xmm1 - movq 16(%rsp), %rax - movq 8(%rsp), %rdx + movq 48(%rsp), %rbp + movupd 32(%rsp), %xmm0 + movupd 16(%rsp), %xmm1 + movq 8(%rsp), %rax + movq 0(%rsp), %rdx addq $56, %rsp - popq %rbp + .cfi_def_cfa_offset 8 retq .Ltmp3: .size __xray_FunctionExit, .Ltmp3-__xray_FunctionExit .cfi_endproc +//===----------------------------------------------------------------------===// + .global __xray_FunctionTailExit .align 16, 0x90 .type __xray_FunctionTailExit,@function @@ -126,8 +139,6 @@ __xray_FunctionTailExit: // this is an exit. In the future, we will introduce a new entry type that // differentiates between a normal exit and a tail exit, but we'd have to do // this and increment the version number for the header. - pushq %rbp - .cfi_def_cfa_offset 16 SAVE_REGISTERS movq _ZN6__xray19XRayPatchedFunctionE(%rip), %rax @@ -140,8 +151,82 @@ __xray_FunctionTailExit: .Ltmp4: RESTORE_REGISTERS - popq %rbp retq .Ltmp5: .size __xray_FunctionTailExit, .Ltmp5-__xray_FunctionTailExit .cfi_endproc + +//===----------------------------------------------------------------------===// + + .globl __xray_ArgLoggerEntry + .align 16, 0x90 + .type __xray_ArgLoggerEntry,@function +__xray_ArgLoggerEntry: + .cfi_startproc + SAVE_REGISTERS + + // Again, these function pointer loads must be atomic; MOV is fine. + movq _ZN6__xray13XRayArgLoggerE(%rip), %rax + testq %rax, %rax + jne .Larg1entryLog + + // If [arg1 logging handler] not set, defer to no-arg logging. + movq _ZN6__xray19XRayPatchedFunctionE(%rip), %rax + testq %rax, %rax + je .Larg1entryFail + +.Larg1entryLog: + + // First argument will become the third + movq %rdi, %rdx + + // XRayEntryType::ENTRY into the second + xorq %rsi, %rsi + + // 32-bit function ID becomes the first + movl %r10d, %edi + callq *%rax + +.Larg1entryFail: + RESTORE_REGISTERS + retq + +.Larg1entryEnd: + .size __xray_ArgLoggerEntry, .Larg1entryEnd-__xray_ArgLoggerEntry + .cfi_endproc + +//===----------------------------------------------------------------------===// + + .global __xray_CustomEvent + .align 16, 0x90 + .type __xray_CustomEvent,@function +__xray_CustomEvent: + .cfi_startproc + subq $16, %rsp + .cfi_def_cfa_offset 24 + movq %rbp, 8(%rsp) + movq %rax, 0(%rsp) + + // We take two arguments to this trampoline, which should be in rdi and rsi + // already. We also make sure that we stash %rax because we use that register + // to call the logging handler. + movq _ZN6__xray22XRayPatchedCustomEventE(%rip), %rax + testq %rax,%rax + je .LcustomEventCleanup + + // At this point we know that rcx and rdx already has the data, so we just + // call the logging handler. + callq *%rax + +.LcustomEventCleanup: + movq 0(%rsp), %rax + movq 8(%rsp), %rbp + addq $16, %rsp + .cfi_def_cfa_offset 8 + retq + +.Ltmp8: + .size __xray_CustomEvent, .Ltmp8-__xray_CustomEvent + .cfi_endproc + +NO_EXEC_STACK_DIRECTIVE diff --git a/contrib/compiler-rt/lib/xray/xray_tsc.h b/contrib/compiler-rt/lib/xray/xray_tsc.h new file mode 100644 index 0000000..4507564 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_tsc.h @@ -0,0 +1,68 @@ +//===-- xray_tsc.h ----------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +//===----------------------------------------------------------------------===// +#ifndef XRAY_EMULATE_TSC_H +#define XRAY_EMULATE_TSC_H + +namespace __xray { +static constexpr uint64_t NanosecondsPerSecond = 1000ULL * 1000 * 1000; +} + +#if defined(__x86_64__) +#include "xray_x86_64.inc" +#elif defined(__powerpc64__) +#include "xray_powerpc64.inc" +#elif defined(__arm__) || defined(__aarch64__) || defined(__mips__) +// Emulated TSC. +// There is no instruction like RDTSCP in user mode on ARM. ARM's CP15 does +// not have a constant frequency like TSC on x86(_64), it may go faster +// or slower depending on CPU turbo or power saving mode. Furthermore, +// to read from CP15 on ARM a kernel modification or a driver is needed. +// We can not require this from users of compiler-rt. +// So on ARM we use clock_gettime() which gives the result in nanoseconds. +// To get the measurements per second, we scale this by the number of +// nanoseconds per second, pretending that the TSC frequency is 1GHz and +// one TSC tick is 1 nanosecond. +#include "sanitizer_common/sanitizer_common.h" +#include "sanitizer_common/sanitizer_internal_defs.h" +#include "xray_defs.h" +#include <cerrno> +#include <cstdint> +#include <time.h> + +namespace __xray { + +inline bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; } + +ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT { + timespec TS; + int result = clock_gettime(CLOCK_REALTIME, &TS); + if (result != 0) { + Report("clock_gettime(2) returned %d, errno=%d.", result, int(errno)); + TS.tv_sec = 0; + TS.tv_nsec = 0; + } + CPU = 0; + return TS.tv_sec * NanosecondsPerSecond + TS.tv_nsec; +} + +inline uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT { + return NanosecondsPerSecond; +} + +} // namespace __xray + +#else +#error Target architecture is not supported. +#endif // CPU architecture + +#endif // XRAY_EMULATE_TSC_H diff --git a/contrib/compiler-rt/lib/xray/xray_utils.cc b/contrib/compiler-rt/lib/xray/xray_utils.cc new file mode 100644 index 0000000..b9a38d1 --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_utils.cc @@ -0,0 +1,125 @@ +//===-- xray_utils.cc -------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +//===----------------------------------------------------------------------===// +#include "xray_utils.h" + +#include "sanitizer_common/sanitizer_common.h" +#include "xray_defs.h" +#include "xray_flags.h" +#include <stdlib.h> +#include <cstdio> +#include <errno.h> +#include <fcntl.h> +#include <iterator> +#include <sys/types.h> +#include <tuple> +#include <unistd.h> +#include <utility> + +namespace __xray { + +void printToStdErr(const char *Buffer) XRAY_NEVER_INSTRUMENT { + fprintf(stderr, "%s", Buffer); +} + +void retryingWriteAll(int Fd, char *Begin, char *End) XRAY_NEVER_INSTRUMENT { + if (Begin == End) + return; + auto TotalBytes = std::distance(Begin, End); + while (auto Written = write(Fd, Begin, TotalBytes)) { + if (Written < 0) { + if (errno == EINTR) + continue; // Try again. + Report("Failed to write; errno = %d\n", errno); + return; + } + TotalBytes -= Written; + if (TotalBytes == 0) + break; + Begin += Written; + } +} + +std::pair<ssize_t, bool> retryingReadSome(int Fd, char *Begin, + char *End) XRAY_NEVER_INSTRUMENT { + auto BytesToRead = std::distance(Begin, End); + ssize_t BytesRead; + ssize_t TotalBytesRead = 0; + while (BytesToRead && (BytesRead = read(Fd, Begin, BytesToRead))) { + if (BytesRead == -1) { + if (errno == EINTR) + continue; + Report("Read error; errno = %d\n", errno); + return std::make_pair(TotalBytesRead, false); + } + + TotalBytesRead += BytesRead; + BytesToRead -= BytesRead; + Begin += BytesRead; + } + return std::make_pair(TotalBytesRead, true); +} + +bool readValueFromFile(const char *Filename, + long long *Value) XRAY_NEVER_INSTRUMENT { + int Fd = open(Filename, O_RDONLY | O_CLOEXEC); + if (Fd == -1) + return false; + static constexpr size_t BufSize = 256; + char Line[BufSize] = {}; + ssize_t BytesRead; + bool Success; + std::tie(BytesRead, Success) = retryingReadSome(Fd, Line, Line + BufSize); + if (!Success) + return false; + close(Fd); + char *End = nullptr; + long long Tmp = internal_simple_strtoll(Line, &End, 10); + bool Result = false; + if (Line[0] != '\0' && (*End == '\n' || *End == '\0')) { + *Value = Tmp; + Result = true; + } + return Result; +} + +int getLogFD() XRAY_NEVER_INSTRUMENT { + // Open a temporary file once for the log. + static char TmpFilename[256] = {}; + static char TmpWildcardPattern[] = "XXXXXX"; + auto Argv = GetArgv(); + const char *Progname = Argv[0] == nullptr ? "(unknown)" : Argv[0]; + const char *LastSlash = internal_strrchr(Progname, '/'); + + if (LastSlash != nullptr) + Progname = LastSlash + 1; + + const int HalfLength = sizeof(TmpFilename) / 2 - sizeof(TmpWildcardPattern); + int NeededLength = internal_snprintf( + TmpFilename, sizeof(TmpFilename), "%.*s%.*s.%s", HalfLength, + flags()->xray_logfile_base, HalfLength, Progname, TmpWildcardPattern); + if (NeededLength > int(sizeof(TmpFilename))) { + Report("XRay log file name too long (%d): %s\n", NeededLength, TmpFilename); + return -1; + } + int Fd = mkstemp(TmpFilename); + if (Fd == -1) { + Report("XRay: Failed opening temporary file '%s'; not logging events.\n", + TmpFilename); + return -1; + } + Report("XRay: Log file in '%s'\n", TmpFilename); + + return Fd; +} + +} // namespace __xray diff --git a/contrib/compiler-rt/lib/xray/xray_utils.h b/contrib/compiler-rt/lib/xray/xray_utils.h new file mode 100644 index 0000000..1ecc74a --- /dev/null +++ b/contrib/compiler-rt/lib/xray/xray_utils.h @@ -0,0 +1,41 @@ +//===-- xray_utils.h --------------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file is a part of XRay, a dynamic runtime instrumentation system. +// +// Some shared utilities for the XRay runtime implementation. +// +//===----------------------------------------------------------------------===// +#ifndef XRAY_UTILS_H +#define XRAY_UTILS_H + +#include <sys/types.h> +#include <utility> + +namespace __xray { + +// Default implementation of the reporting interface for sanitizer errors. +void printToStdErr(const char *Buffer); + +// EINTR-safe write routine, provided a file descriptor and a character range. +void retryingWriteAll(int Fd, char *Begin, char *End); + +// Reads a long long value from a provided file. +bool readValueFromFile(const char *Filename, long long *Value); + +// EINTR-safe read routine, providing a file descriptor and a character range. +std::pair<ssize_t, bool> retryingReadSome(int Fd, char *Begin, char *End); + +// EINTR-safe open routine, uses flag-provided values for initialising a log +// file. +int getLogFD(); + +} // namespace __xray + +#endif // XRAY_UTILS_H diff --git a/contrib/compiler-rt/lib/xray/xray_x86_64.cc b/contrib/compiler-rt/lib/xray/xray_x86_64.cc index 3ee9189..e34806f 100644 --- a/contrib/compiler-rt/lib/xray/xray_x86_64.cc +++ b/contrib/compiler-rt/lib/xray/xray_x86_64.cc @@ -1,6 +1,8 @@ +#include "cpuid.h" #include "sanitizer_common/sanitizer_common.h" #include "xray_defs.h" #include "xray_interface_internal.h" + #include <atomic> #include <cstdint> #include <errno.h> @@ -42,9 +44,9 @@ static bool readValueFromFile(const char *Filename, ssize_t BytesRead; bool Success; std::tie(BytesRead, Success) = retryingReadSome(Fd, Line, Line + BufSize); + close(Fd); if (!Success) return false; - close(Fd); char *End = nullptr; long long Tmp = internal_simple_strtoll(Line, &End, 10); bool Result = false; @@ -55,32 +57,35 @@ static bool readValueFromFile(const char *Filename, return Result; } -uint64_t cycleFrequency() XRAY_NEVER_INSTRUMENT { - long long CPUFrequency = -1; +uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT { + long long TSCFrequency = -1; if (readValueFromFile("/sys/devices/system/cpu/cpu0/tsc_freq_khz", - &CPUFrequency)) { - CPUFrequency *= 1000; + &TSCFrequency)) { + TSCFrequency *= 1000; } else if (readValueFromFile( - "/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq", - &CPUFrequency)) { - CPUFrequency *= 1000; + "/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq", + &TSCFrequency)) { + TSCFrequency *= 1000; } else { Report("Unable to determine CPU frequency for TSC accounting.\n"); } - return CPUFrequency == -1 ? 0 : static_cast<uint64_t>(CPUFrequency); + return TSCFrequency == -1 ? 0 : static_cast<uint64_t>(TSCFrequency); } static constexpr uint8_t CallOpCode = 0xe8; static constexpr uint16_t MovR10Seq = 0xba41; static constexpr uint16_t Jmp9Seq = 0x09eb; +static constexpr uint16_t Jmp20Seq = 0x14eb; static constexpr uint8_t JmpOpCode = 0xe9; static constexpr uint8_t RetOpCode = 0xc3; +static constexpr uint16_t NopwSeq = 0x9066; static constexpr int64_t MinOffset{std::numeric_limits<int32_t>::min()}; static constexpr int64_t MaxOffset{std::numeric_limits<int32_t>::max()}; bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, - const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + const XRaySledEntry &Sled, + void (*Trampoline)()) XRAY_NEVER_INSTRUMENT { // Here we do the dance of replacing the following sled: // // xray_sled_n: @@ -101,13 +106,12 @@ bool patchFunctionEntry(const bool Enable, const uint32_t FuncId, // 4. Do an atomic write over the jmp instruction for the "mov r10d" // opcode and first operand. // - // Prerequisite is to compute the relative offset to the - // __xray_FunctionEntry function's address. - int64_t TrampolineOffset = reinterpret_cast<int64_t>(__xray_FunctionEntry) - + // Prerequisite is to compute the relative offset to the trampoline's address. + int64_t TrampolineOffset = reinterpret_cast<int64_t>(Trampoline) - (static_cast<int64_t>(Sled.Address) + 11); if (TrampolineOffset < MinOffset || TrampolineOffset > MaxOffset) { Report("XRay Entry trampoline (%p) too far from sled (%p)\n", - __xray_FunctionEntry, reinterpret_cast<void *>(Sled.Address)); + Trampoline, reinterpret_cast<void *>(Sled.Address)); return false; } if (Enable) { @@ -199,4 +203,60 @@ bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId, return true; } +bool patchCustomEvent(const bool Enable, const uint32_t FuncId, + const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT { + // Here we do the dance of replacing the following sled: + // + // xray_sled_n: + // jmp +19 // 2 bytes + // ... + // + // With the following: + // + // nopw // 2 bytes* + // ... + // + // We need to do this in the following order: + // + // 1. Overwrite the 5-byte nop with the call (relative), where (relative) is + // the relative offset to the __xray_CustomEvent trampoline. + // 2. Do a two-byte atomic write over the 'jmp +24' to turn it into a 'nopw'. + // This allows us to "enable" this code once the changes have committed. + // + // The "unpatch" should just turn the 'nopw' back to a 'jmp +24'. + // + if (Enable) { + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), NopwSeq, + std::memory_order_release); + } else { + std::atomic_store_explicit( + reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), Jmp20Seq, + std::memory_order_release); + } + return false; +} + +// We determine whether the CPU we're running on has the correct features we +// need. In x86_64 this will be rdtscp support. +bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { + unsigned int EAX, EBX, ECX, EDX; + + // We check whether rdtscp support is enabled. According to the x86_64 manual, + // level should be set at 0x80000001, and we should have a look at bit 27 in + // EDX. That's 0x8000000 (or 1u << 26). + __get_cpuid(0x80000001, &EAX, &EBX, &ECX, &EDX); + if (!(EDX & (1u << 26))) { + Report("Missing rdtscp support.\n"); + return false; + } + // Also check whether we can determine the CPU frequency, since if we cannot, + // we should use the emulated TSC instead. + if (!getTSCFrequency()) { + Report("Unable to determine CPU frequency.\n"); + return false; + } + return true; +} + } // namespace __xray diff --git a/contrib/compiler-rt/lib/xray/xray_x86_64.h b/contrib/compiler-rt/lib/xray/xray_x86_64.inc index 52d2dea..4ad3f98 100644 --- a/contrib/compiler-rt/lib/xray/xray_x86_64.h +++ b/contrib/compiler-rt/lib/xray/xray_x86_64.inc @@ -1,4 +1,4 @@ -//===-- xray_x86_64.h -------------------------------------------*- C++ -*-===// +//===-- xray_x86_64.inc -----------------------------------------*- C++ -*-===// // // The LLVM Compiler Infrastructure // @@ -10,8 +10,6 @@ // This file is a part of XRay, a dynamic runtime instrumentation system. // //===----------------------------------------------------------------------===// -#ifndef XRAY_X86_64_H -#define XRAY_X86_64_H #include <cstdint> #include <x86intrin.h> @@ -27,6 +25,9 @@ ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT { CPU = LongCPU; return TSC; } -} -#endif // XRAY_X86_64_H +uint64_t getTSCFrequency(); + +bool probeRequiredCPUFeatures(); + +} // namespace __xray |
