diff options
author | dim <dim@FreeBSD.org> | 2017-09-26 19:56:36 +0000 |
---|---|---|
committer | Luiz Souza <luiz@netgate.com> | 2018-02-21 15:12:19 -0300 |
commit | 1dcd2e8d24b295bc73e513acec2ed1514bb66be4 (patch) | |
tree | 4bd13a34c251e980e1a6b13584ca1f63b0dfe670 /contrib/llvm/lib/ProfileData | |
parent | f45541ca2a56a1ba1202f94c080b04e96c1fa239 (diff) | |
download | FreeBSD-src-1dcd2e8d24b295bc73e513acec2ed1514bb66be4.zip FreeBSD-src-1dcd2e8d24b295bc73e513acec2ed1514bb66be4.tar.gz |
Merge clang, llvm, lld, lldb, compiler-rt and libc++ 5.0.0 release.
MFC r309126 (by emaste):
Correct lld llvm-tblgen dependency file name
MFC r309169:
Get rid of separate Subversion mergeinfo properties for llvm-dwarfdump
and llvm-lto. The mergeinfo confuses Subversion enormously, and these
directories will just use the mergeinfo for llvm itself.
MFC r312765:
Pull in r276136 from upstream llvm trunk (by Wei Mi):
Use ValueOffsetPair to enhance value reuse during SCEV expansion.
In D12090, the ExprValueMap was added to reuse existing value during
SCEV expansion. However, const folding and sext/zext distribution can
make the reuse still difficult.
A simplified case is: suppose we know S1 expands to V1 in
ExprValueMap, and
S1 = S2 + C_a
S3 = S2 + C_b
where C_a and C_b are different SCEVConstants. Then we'd like to
expand S3 as V1 - C_a + C_b instead of expanding S2 literally. It is
helpful when S2 is a complex SCEV expr and S2 has no entry in
ExprValueMap, which is usually caused by the fact that S3 is
generated from S1 after const folding.
In order to do that, we represent ExprValueMap as a mapping from SCEV
to ValueOffsetPair. We will save both S1->{V1, 0} and S2->{V1, C_a}
into the ExprValueMap when we create SCEV for V1. When S3 is
expanded, it will first expand S2 to V1 - C_a because of S2->{V1,
C_a} in the map, then expand S3 to V1 - C_a + C_b.
Differential Revision: https://reviews.llvm.org/D21313
This should fix assertion failures when building OpenCV >= 3.1.
PR: 215649
MFC r312831:
Revert r312765 for now, since it causes assertions when building
lang/spidermonkey24.
Reported by: antoine
PR: 215649
MFC r316511 (by jhb):
Add an implementation of __ffssi2() derived from __ffsdi2().
Newer versions of GCC include an __ffssi2() symbol in libgcc and the
compiler can emit calls to it in generated code. This is true for at
least GCC 6.2 when compiling world for mips and mips64.
Reviewed by: jmallett, dim
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D10086
MFC r318601 (by adrian):
[libcompiler-rt] add bswapdi2/bswapsi2
This is required for mips gcc 6.3 userland to build/run.
Reviewed by: emaste, dim
Approved by: emaste
Differential Revision: https://reviews.freebsd.org/D10838
MFC r318884 (by emaste):
lldb: map TRAP_CAP to a trace trap
In the absense of a more specific handler for TRAP_CAP (generated by
ENOTCAPABLE or ECAPMODE while in capability mode) treat it as a trace
trap.
Example usage (testing the bug in PR219173):
% proccontrol -m trapcap lldb usr.bin/hexdump/obj/hexdump -- -Cv -s 1 /bin/ls
...
(lldb) run
Process 12980 launching
Process 12980 launched: '.../usr.bin/hexdump/obj/hexdump' (x86_64)
Process 12980 stopped
* thread #1, stop reason = trace
frame #0: 0x0000004b80c65f1a libc.so.7`__sys_lseek + 10
...
In the future we should have LLDB control the trapcap procctl itself
(as it does with ASLR), as well as report a specific stop reason.
This change eliminates an assertion failure from LLDB for now.
MFC r319796:
Remove a few unneeded files from libllvm, libclang and liblldb.
MFC r319885 (by emaste):
lld: ELF: Fix ICF crash on absolute symbol relocations.
If two sections contained relocations to absolute symbols with the same
value we would crash when trying to access their sections. Add a check that
both symbols point to sections before accessing their sections, and treat
absolute symbols as equal if their values are equal.
Obtained from: LLD commit r292578
MFC r319918:
Revert r319796 for now, it can cause undefined references when linking
in some circumstances.
Reported by: Shawn Webb <shawn.webb@hardenedbsd.org>
MFC r319957 (by emaste):
lld: Add armelf emulation mode
Obtained from: LLD r305375
MFC r321369:
Upgrade our copies of clang, llvm, lld, lldb, compiler-rt and libc++ to
5.0.0 (trunk r308421). Upstream has branched for the 5.0.0 release,
which should be in about a month. Please report bugs and regressions,
so we can get them into the release.
Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.
MFC r321420:
Add a few more object files to liblldb, which should solve errors when
linking the lldb executable in some cases. In particular, when the
-ffunction-sections -fdata-sections options are turned off, or
ineffective.
Reported by: Shawn Webb, Mark Millard
MFC r321433:
Cleanup stale Options.inc files from the previous libllvm build for
clang 4.0.0. Otherwise, these can get included before the two newly
generated ones (which are different) for clang 5.0.0.
Reported by: Mark Millard
MFC r321439 (by bdrewery):
Move llvm Options.inc hack from r321433 for NO_CLEAN to lib/clang/libllvm.
The files are only ever generated to .OBJDIR, not to WORLDTMP (as a
sysroot) and are only ever included from a compilation. So using
a beforebuild target here removes the file before the compilation
tries to include it.
MFC r321664:
Pull in r308891 from upstream llvm trunk (by Benjamin Kramer):
[CodeGenPrepare] Cut off FindAllMemoryUses if there are too many uses.
This avoids excessive compile time. The case I'm looking at is
Function.cpp from an old version of LLVM that still had the giant
memcmp string matcher in it. Before r308322 this compiled in about 2
minutes, after it, clang takes infinite* time to compile it. With
this patch we're at 5 min, which is still bad but this is a
pathological case.
The cut off at 20 uses was chosen by looking at other cut-offs in LLVM
for user scanning. It's probably too high, but does the job and is
very unlikely to regress anything.
Fixes PR33900.
* I'm impatient and aborted after 15 minutes, on the bug report it was
killed after 2h.
Pull in r308986 from upstream llvm trunk (by Simon Pilgrim):
[X86][CGP] Reduce memcmp() expansion to 2 load pairs (PR33914)
D35067/rL308322 attempted to support up to 4 load pairs for memcmp
inlining which resulted in regressions for some optimized libc memcmp
implementations (PR33914).
Until we can match these more optimal cases, this patch reduces the
memcmp expansion to a maximum of 2 load pairs (which matches what we
do for -Os).
This patch should be considered for the 5.0.0 release branch as well
Differential Revision: https://reviews.llvm.org/D35830
These fix a hang (or extremely long compile time) when building older
LLVM ports.
Reported by: antoine
PR: 219139
MFC r321719:
Pull in r309503 from upstream clang trunk (by Richard Smith):
PR33902: Invalidate line number cache when adding more text to
existing buffer.
This led to crashes as the line number cache would report a bogus
line number for a line of code, and we'd try to find a nonexistent
column within the line when printing diagnostics.
This fixes an assertion when building the graphics/champlain port.
Reported by: antoine, kwm
PR: 219139
MFC r321723:
Upgrade our copies of clang, llvm, lld and lldb to r309439 from the
upstream release_50 branch. This is just after upstream's 5.0.0-rc1.
MFC r322320:
Upgrade our copies of clang, llvm and libc++ to r310316 from the
upstream release_50 branch.
MFC r322326 (by emaste):
lldb: Make i386-*-freebsd expression work on JIT path
* Enable i386 ABI creation for freebsd
* Added an extra argument in ABISysV_i386::PrepareTrivialCall for mmap
syscall
* Unlike linux, the last argument of mmap is actually 64-bit(off_t).
This requires us to push an additional word for the higher order bits.
* Prior to this change, ktrace dump will show mmap failures due to
invalid argument coming from the 6th mmap argument.
Submitted by: Karnajit Wangkhem
Differential Revision: https://reviews.llvm.org/D34776
MFC r322360 (by emaste):
lldb: Report inferior signals as signals, not exceptions, on FreeBSD
This is the FreeBSD equivalent of LLVM r238549.
This serves 2 purposes:
* LLDB should handle inferior process signals SIGSEGV/SIGILL/SIGBUS/
SIGFPE the way it is suppose to be handled. Prior to this fix these
signals will neither create a coredump, nor exit from the debugger
or work for signal handling scenario.
* eInvalidCrashReason need not report "unknown crash reason" if we have
a valid si_signo
llvm.org/pr23699
Patch by Karnajit Wangkhem
Differential Revision: https://reviews.llvm.org/D35223
Submitted by: Karnajit Wangkhem
Obtained from: LLVM r310591
MFC r322474 (by emaste):
lld: Add `-z muldefs` option.
Obtained from: LLVM r310757
MFC r322740:
Upgrade our copies of clang, llvm, lld and libc++ to r311219 from the
upstream release_50 branch.
MFC r322855:
Upgrade our copies of clang, llvm, lldb and compiler-rt to r311606 from
the upstream release_50 branch.
As of this version, lib/msun's trig test should also work correctly
again (see bug 220989 for more information).
PR: 220989
MFC r323112:
Upgrade our copies of clang, llvm, lldb and compiler-rt to r312293 from
the upstream release_50 branch. This corresponds to 5.0.0 rc4.
As of this version, the cad/stepcode port should now compile in a more
reasonable time on i386 (see bug 221836 for more information).
PR: 221836
MFC r323245:
Upgrade our copies of clang, llvm, lld, lldb, compiler-rt and libc++ to
5.0.0 release (upstream r312559).
Release notes for llvm, clang and lld will be available here soon:
<http://releases.llvm.org/5.0.0/docs/ReleaseNotes.html>
<http://releases.llvm.org/5.0.0/tools/clang/docs/ReleaseNotes.html>
<http://releases.llvm.org/5.0.0/tools/lld/docs/ReleaseNotes.html>
Relnotes: yes
(cherry picked from commit 12cd91cf4c6b96a24427c0de5374916f2808d263)
Diffstat (limited to 'contrib/llvm/lib/ProfileData')
-rw-r--r-- | contrib/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp | 113 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp | 86 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp | 35 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/InstrProf.cpp | 250 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/InstrProfReader.cpp | 83 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/InstrProfWriter.cpp | 102 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/SampleProf.cpp | 34 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/SampleProfReader.cpp | 31 | ||||
-rw-r--r-- | contrib/llvm/lib/ProfileData/SampleProfWriter.cpp | 112 |
9 files changed, 580 insertions, 266 deletions
diff --git a/contrib/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp b/contrib/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp index 6d907c7..8c5f136 100644 --- a/contrib/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp +++ b/contrib/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp @@ -1,4 +1,4 @@ -//=-- CoverageMapping.cpp - Code coverage mapping support ---------*- C++ -*-=// +//===- CoverageMapping.cpp - Code coverage mapping support ----------------===// // // The LLVM Compiler Infrastructure // @@ -13,17 +13,31 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/Coverage/CoverageMapping.h" +#include "llvm/ADT/ArrayRef.h" #include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/None.h" #include "llvm/ADT/Optional.h" #include "llvm/ADT/SmallBitVector.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" #include "llvm/ProfileData/Coverage/CoverageMappingReader.h" #include "llvm/ProfileData/InstrProfReader.h" #include "llvm/Support/Debug.h" #include "llvm/Support/Errc.h" +#include "llvm/Support/Error.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ManagedStatic.h" -#include "llvm/Support/Path.h" +#include "llvm/Support/MemoryBuffer.h" #include "llvm/Support/raw_ostream.h" +#include <algorithm> +#include <cassert> +#include <cstdint> +#include <iterator> +#include <memory> +#include <string> +#include <system_error> +#include <utility> +#include <vector> using namespace llvm; using namespace coverage; @@ -40,26 +54,26 @@ Counter CounterExpressionBuilder::get(const CounterExpression &E) { return Counter::getExpression(I); } -void CounterExpressionBuilder::extractTerms( - Counter C, int Sign, SmallVectorImpl<std::pair<unsigned, int>> &Terms) { +void CounterExpressionBuilder::extractTerms(Counter C, int Factor, + SmallVectorImpl<Term> &Terms) { switch (C.getKind()) { case Counter::Zero: break; case Counter::CounterValueReference: - Terms.push_back(std::make_pair(C.getCounterID(), Sign)); + Terms.emplace_back(C.getCounterID(), Factor); break; case Counter::Expression: const auto &E = Expressions[C.getExpressionID()]; - extractTerms(E.LHS, Sign, Terms); - extractTerms(E.RHS, E.Kind == CounterExpression::Subtract ? -Sign : Sign, - Terms); + extractTerms(E.LHS, Factor, Terms); + extractTerms( + E.RHS, E.Kind == CounterExpression::Subtract ? -Factor : Factor, Terms); break; } } Counter CounterExpressionBuilder::simplify(Counter ExpressionTree) { // Gather constant terms. - llvm::SmallVector<std::pair<unsigned, int>, 32> Terms; + SmallVector<Term, 32> Terms; extractTerms(ExpressionTree, +1, Terms); // If there are no terms, this is just a zero. The algorithm below assumes at @@ -68,17 +82,15 @@ Counter CounterExpressionBuilder::simplify(Counter ExpressionTree) { return Counter::getZero(); // Group the terms by counter ID. - std::sort(Terms.begin(), Terms.end(), - [](const std::pair<unsigned, int> &LHS, - const std::pair<unsigned, int> &RHS) { - return LHS.first < RHS.first; + std::sort(Terms.begin(), Terms.end(), [](const Term &LHS, const Term &RHS) { + return LHS.CounterID < RHS.CounterID; }); // Combine terms by counter ID to eliminate counters that sum to zero. auto Prev = Terms.begin(); for (auto I = Prev + 1, E = Terms.end(); I != E; ++I) { - if (I->first == Prev->first) { - Prev->second += I->second; + if (I->CounterID == Prev->CounterID) { + Prev->Factor += I->Factor; continue; } ++Prev; @@ -89,24 +101,24 @@ Counter CounterExpressionBuilder::simplify(Counter ExpressionTree) { Counter C; // Create additions. We do this before subtractions to avoid constructs like // ((0 - X) + Y), as opposed to (Y - X). - for (auto Term : Terms) { - if (Term.second <= 0) + for (auto T : Terms) { + if (T.Factor <= 0) continue; - for (int I = 0; I < Term.second; ++I) + for (int I = 0; I < T.Factor; ++I) if (C.isZero()) - C = Counter::getCounter(Term.first); + C = Counter::getCounter(T.CounterID); else C = get(CounterExpression(CounterExpression::Add, C, - Counter::getCounter(Term.first))); + Counter::getCounter(T.CounterID))); } // Create subtractions. - for (auto Term : Terms) { - if (Term.second >= 0) + for (auto T : Terms) { + if (T.Factor >= 0) continue; - for (int I = 0; I < -Term.second; ++I) + for (int I = 0; I < -T.Factor; ++I) C = get(CounterExpression(CounterExpression::Subtract, C, - Counter::getCounter(Term.first))); + Counter::getCounter(T.CounterID))); } return C; } @@ -120,8 +132,7 @@ Counter CounterExpressionBuilder::subtract(Counter LHS, Counter RHS) { get(CounterExpression(CounterExpression::Subtract, LHS, RHS))); } -void CounterMappingContext::dump(const Counter &C, - llvm::raw_ostream &OS) const { +void CounterMappingContext::dump(const Counter &C, raw_ostream &OS) const { switch (C.getKind()) { case Counter::Zero: OS << '0'; @@ -145,7 +156,7 @@ void CounterMappingContext::dump(const Counter &C, return; Expected<int64_t> Value = evaluate(C); if (auto E = Value.takeError()) { - llvm::consumeError(std::move(E)); + consumeError(std::move(E)); return; } OS << '[' << *Value << ']'; @@ -187,6 +198,9 @@ Error CoverageMapping::loadFunctionRecord( const CoverageMappingRecord &Record, IndexedInstrProfReader &ProfileReader) { StringRef OrigFuncName = Record.FunctionName; + if (OrigFuncName.empty()) + return make_error<CoverageMapError>(coveragemap_error::malformed); + if (Record.Filenames.empty()) OrigFuncName = getFuncNameWithoutPrefix(OrigFuncName); else @@ -217,7 +231,7 @@ Error CoverageMapping::loadFunctionRecord( for (const auto &Region : Record.MappingRegions) { Expected<int64_t> ExecutionCount = Ctx.evaluate(Region.Count); if (auto E = ExecutionCount.takeError()) { - llvm::consumeError(std::move(E)); + consumeError(std::move(E)); return Error::success(); } Function.pushRegion(Region, *ExecutionCount); @@ -231,18 +245,6 @@ Error CoverageMapping::loadFunctionRecord( return Error::success(); } -Expected<std::unique_ptr<CoverageMapping>> -CoverageMapping::load(CoverageMappingReader &CoverageReader, - IndexedInstrProfReader &ProfileReader) { - auto Coverage = std::unique_ptr<CoverageMapping>(new CoverageMapping()); - - for (const auto &Record : CoverageReader) - if (Error E = Coverage->loadFunctionRecord(Record, ProfileReader)) - return std::move(E); - - return std::move(Coverage); -} - Expected<std::unique_ptr<CoverageMapping>> CoverageMapping::load( ArrayRef<std::unique_ptr<CoverageMappingReader>> CoverageReaders, IndexedInstrProfReader &ProfileReader) { @@ -281,13 +283,14 @@ CoverageMapping::load(ArrayRef<StringRef> ObjectFilenames, } namespace { + /// \brief Distributes functions into instantiation sets. /// /// An instantiation set is a collection of functions that have the same source /// code, ie, template functions specializations. class FunctionInstantiationSetCollector { - typedef DenseMap<std::pair<unsigned, unsigned>, - std::vector<const FunctionRecord *>> MapT; + using MapT = DenseMap<std::pair<unsigned, unsigned>, + std::vector<const FunctionRecord *>>; MapT InstantiatedFunctions; public: @@ -301,7 +304,6 @@ public: } MapT::iterator begin() { return InstantiatedFunctions.begin(); } - MapT::iterator end() { return InstantiatedFunctions.end(); } }; @@ -326,7 +328,7 @@ class SegmentBuilder { Segments.pop_back(); DEBUG(dbgs() << "Segment at " << Line << ":" << Col); // Set this region's count. - if (Region.Kind != coverage::CounterMappingRegion::SkippedRegion) { + if (Region.Kind != CounterMappingRegion::SkippedRegion) { DEBUG(dbgs() << " with count " << Region.ExecutionCount); Segments.emplace_back(Line, Col, Region.ExecutionCount, IsRegionEntry); } else @@ -380,10 +382,10 @@ class SegmentBuilder { // in combineRegions(). Because we accumulate counter values only from // regions of the same kind as the first region of the area, prefer // CodeRegion to ExpansionRegion and ExpansionRegion to SkippedRegion. - static_assert(coverage::CounterMappingRegion::CodeRegion < - coverage::CounterMappingRegion::ExpansionRegion && - coverage::CounterMappingRegion::ExpansionRegion < - coverage::CounterMappingRegion::SkippedRegion, + static_assert(CounterMappingRegion::CodeRegion < + CounterMappingRegion::ExpansionRegion && + CounterMappingRegion::ExpansionRegion < + CounterMappingRegion::SkippedRegion, "Unexpected order of region kind values"); return LHS.Kind < RHS.Kind; }); @@ -437,7 +439,8 @@ public: return Segments; } }; -} + +} // end anonymous namespace std::vector<StringRef> CoverageMapping::getUniqueSourceFiles() const { std::vector<StringRef> Filenames; @@ -487,7 +490,7 @@ static bool isExpansion(const CountedRegion &R, unsigned FileID) { CoverageData CoverageMapping::getCoverageForFile(StringRef Filename) const { CoverageData FileCoverage(Filename); - std::vector<coverage::CountedRegion> Regions; + std::vector<CountedRegion> Regions; for (const auto &Function : Functions) { auto MainFileID = findMainViewFileID(Filename, Function); @@ -533,7 +536,7 @@ CoverageMapping::getCoverageForFunction(const FunctionRecord &Function) const { return CoverageData(); CoverageData FunctionCoverage(Function.Filenames[*MainFileID]); - std::vector<coverage::CountedRegion> Regions; + std::vector<CountedRegion> Regions; for (const auto &CR : Function.CountedRegions) if (CR.FileID == *MainFileID) { Regions.push_back(CR); @@ -551,7 +554,7 @@ CoverageData CoverageMapping::getCoverageForExpansion( const ExpansionRecord &Expansion) const { CoverageData ExpansionCoverage( Expansion.Function.Filenames[Expansion.FileID]); - std::vector<coverage::CountedRegion> Regions; + std::vector<CountedRegion> Regions; for (const auto &CR : Expansion.Function.CountedRegions) if (CR.FileID == Expansion.FileID) { Regions.push_back(CR); @@ -566,8 +569,7 @@ CoverageData CoverageMapping::getCoverageForExpansion( return ExpansionCoverage; } -namespace { -std::string getCoverageMapErrString(coveragemap_error Err) { +static std::string getCoverageMapErrString(coveragemap_error Err) { switch (Err) { case coveragemap_error::success: return "Success"; @@ -585,6 +587,8 @@ std::string getCoverageMapErrString(coveragemap_error Err) { llvm_unreachable("A value of coveragemap_error has no message."); } +namespace { + // FIXME: This class is only here to support the transition to llvm::Error. It // will be removed once this transition is complete. Clients should prefer to // deal with the Error value directly, rather than converting to error_code. @@ -594,6 +598,7 @@ class CoverageMappingErrorCategoryType : public std::error_category { return getCoverageMapErrString(static_cast<coveragemap_error>(IE)); } }; + } // end anonymous namespace std::string CoverageMapError::message() const { diff --git a/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp b/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp index a6c7031..fff0a03 100644 --- a/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp +++ b/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp @@ -1,4 +1,4 @@ -//=-- CoverageMappingReader.cpp - Code coverage mapping reader ----*- C++ -*-=// +//===- CoverageMappingReader.cpp - Code coverage mapping reader -----------===// // // The LLVM Compiler Infrastructure // @@ -13,14 +13,34 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/Coverage/CoverageMappingReader.h" +#include "llvm/ADT/ArrayRef.h" #include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/STLExtras.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/ADT/Triple.h" +#include "llvm/Object/Binary.h" +#include "llvm/Object/COFF.h" +#include "llvm/Object/Error.h" #include "llvm/Object/MachOUniversal.h" #include "llvm/Object/ObjectFile.h" +#include "llvm/ProfileData/InstrProf.h" +#include "llvm/Support/Casting.h" #include "llvm/Support/Debug.h" #include "llvm/Support/Endian.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/ErrorHandling.h" #include "llvm/Support/LEB128.h" #include "llvm/Support/MathExtras.h" #include "llvm/Support/raw_ostream.h" +#include <algorithm> +#include <cassert> +#include <cstddef> +#include <cstdint> +#include <limits> +#include <memory> +#include <utility> +#include <vector> using namespace llvm; using namespace coverage; @@ -42,7 +62,7 @@ void CoverageMappingIterator::increment() { } Error RawCoverageReader::readULEB128(uint64_t &Result) { - if (Data.size() < 1) + if (Data.empty()) return make_error<CoverageMapError>(coveragemap_error::truncated); unsigned N = 0; Result = decodeULEB128(reinterpret_cast<const uint8_t *>(Data.data()), &N); @@ -226,9 +246,8 @@ Error RawCoverageMappingReader::readMappingRegionsSubArray( } Error RawCoverageMappingReader::read() { - // Read the virtual file mapping. - llvm::SmallVector<unsigned, 8> VirtualFileMapping; + SmallVector<unsigned, 8> VirtualFileMapping; uint64_t NumFileMappings; if (auto Err = readSize(NumFileMappings)) return Err; @@ -349,7 +368,10 @@ static Expected<bool> isCoverageMappingDummy(uint64_t Hash, StringRef Mapping) { } namespace { + struct CovMapFuncRecordReader { + virtual ~CovMapFuncRecordReader() = default; + // The interface to read coverage mapping function records for a module. // // \p Buf points to the buffer containing the \c CovHeader of the coverage @@ -359,26 +381,24 @@ struct CovMapFuncRecordReader { // greater than \p End if not. virtual Expected<const char *> readFunctionRecords(const char *Buf, const char *End) = 0; - virtual ~CovMapFuncRecordReader() {} + template <class IntPtrT, support::endianness Endian> static Expected<std::unique_ptr<CovMapFuncRecordReader>> - get(coverage::CovMapVersion Version, InstrProfSymtab &P, + get(CovMapVersion Version, InstrProfSymtab &P, std::vector<BinaryCoverageReader::ProfileMappingRecord> &R, std::vector<StringRef> &F); }; // A class for reading coverage mapping function records for a module. -template <coverage::CovMapVersion Version, class IntPtrT, - support::endianness Endian> +template <CovMapVersion Version, class IntPtrT, support::endianness Endian> class VersionedCovMapFuncRecordReader : public CovMapFuncRecordReader { - typedef typename coverage::CovMapTraits< - Version, IntPtrT>::CovMapFuncRecordType FuncRecordType; - typedef typename coverage::CovMapTraits<Version, IntPtrT>::NameRefType - NameRefType; + using FuncRecordType = + typename CovMapTraits<Version, IntPtrT>::CovMapFuncRecordType; + using NameRefType = typename CovMapTraits<Version, IntPtrT>::NameRefType; // Maps function's name references to the indexes of their records // in \c Records. - llvm::DenseMap<NameRefType, size_t> FunctionRecords; + DenseMap<NameRefType, size_t> FunctionRecords; InstrProfSymtab &ProfileNames; std::vector<StringRef> &Filenames; std::vector<BinaryCoverageReader::ProfileMappingRecord> &Records; @@ -399,6 +419,8 @@ class VersionedCovMapFuncRecordReader : public CovMapFuncRecordReader { StringRef FuncName; if (Error Err = CFR->template getFuncName<Endian>(ProfileNames, FuncName)) return Err; + if (FuncName.empty()) + return make_error<InstrProfError>(instrprof_error::malformed); Records.emplace_back(Version, FuncName, FuncHash, Mapping, FilenamesBegin, Filenames.size() - FilenamesBegin); return Error::success(); @@ -432,14 +454,16 @@ public: std::vector<BinaryCoverageReader::ProfileMappingRecord> &R, std::vector<StringRef> &F) : ProfileNames(P), Filenames(F), Records(R) {} - ~VersionedCovMapFuncRecordReader() override {} + + ~VersionedCovMapFuncRecordReader() override = default; Expected<const char *> readFunctionRecords(const char *Buf, const char *End) override { using namespace support; + if (Buf + sizeof(CovMapHeader) > End) return make_error<CoverageMapError>(coveragemap_error::malformed); - auto CovHeader = reinterpret_cast<const coverage::CovMapHeader *>(Buf); + auto CovHeader = reinterpret_cast<const CovMapHeader *>(Buf); uint32_t NRecords = CovHeader->getNRecords<Endian>(); uint32_t FilenamesSize = CovHeader->getFilenamesSize<Endian>(); uint32_t CoverageSize = CovHeader->getCoverageSize<Endian>(); @@ -490,14 +514,16 @@ public: return Buf; } }; + } // end anonymous namespace template <class IntPtrT, support::endianness Endian> Expected<std::unique_ptr<CovMapFuncRecordReader>> CovMapFuncRecordReader::get( - coverage::CovMapVersion Version, InstrProfSymtab &P, + CovMapVersion Version, InstrProfSymtab &P, std::vector<BinaryCoverageReader::ProfileMappingRecord> &R, std::vector<StringRef> &F) { using namespace coverage; + switch (Version) { case CovMapVersion::Version1: return llvm::make_unique<VersionedCovMapFuncRecordReader< @@ -518,11 +544,12 @@ static Error readCoverageMappingData( std::vector<BinaryCoverageReader::ProfileMappingRecord> &Records, std::vector<StringRef> &Filenames) { using namespace coverage; + // Read the records in the coverage data section. auto CovHeader = - reinterpret_cast<const coverage::CovMapHeader *>(Data.data()); + reinterpret_cast<const CovMapHeader *>(Data.data()); CovMapVersion Version = (CovMapVersion)CovHeader->getVersion<Endian>(); - if (Version > coverage::CovMapVersion::CurrentVersion) + if (Version > CovMapVersion::CurrentVersion) return make_error<CoverageMapError>(coveragemap_error::unsupported_version); Expected<std::unique_ptr<CovMapFuncRecordReader>> ReaderExpected = CovMapFuncRecordReader::get<T, Endian>(Version, ProfileNames, Records, @@ -538,6 +565,7 @@ static Error readCoverageMappingData( } return Error::success(); } + static const char *TestingFormatMagic = "llvmcovmtestdata"; static Error loadTestingFormat(StringRef Data, InstrProfSymtab &ProfileNames, @@ -548,7 +576,7 @@ static Error loadTestingFormat(StringRef Data, InstrProfSymtab &ProfileNames, Endian = support::endianness::little; Data = Data.substr(StringRef(TestingFormatMagic).size()); - if (Data.size() < 1) + if (Data.empty()) return make_error<CoverageMapError>(coveragemap_error::truncated); unsigned N = 0; auto ProfileNamesSize = @@ -556,7 +584,7 @@ static Error loadTestingFormat(StringRef Data, InstrProfSymtab &ProfileNames, if (N > Data.size()) return make_error<CoverageMapError>(coveragemap_error::malformed); Data = Data.substr(N); - if (Data.size() < 1) + if (Data.empty()) return make_error<CoverageMapError>(coveragemap_error::truncated); N = 0; uint64_t Address = @@ -570,7 +598,7 @@ static Error loadTestingFormat(StringRef Data, InstrProfSymtab &ProfileNames, return E; CoverageMapping = Data.substr(ProfileNamesSize); // Skip the padding bytes because coverage map data has an alignment of 8. - if (CoverageMapping.size() < 1) + if (CoverageMapping.empty()) return make_error<CoverageMapError>(coveragemap_error::truncated); size_t Pad = alignmentAdjustment(CoverageMapping.data(), 8); if (CoverageMapping.size() < Pad) @@ -595,21 +623,21 @@ static Error loadBinaryFormat(MemoryBufferRef ObjectBuffer, StringRef &CoverageMapping, uint8_t &BytesInAddress, support::endianness &Endian, StringRef Arch) { - auto BinOrErr = object::createBinary(ObjectBuffer); + auto BinOrErr = createBinary(ObjectBuffer); if (!BinOrErr) return BinOrErr.takeError(); auto Bin = std::move(BinOrErr.get()); std::unique_ptr<ObjectFile> OF; - if (auto *Universal = dyn_cast<object::MachOUniversalBinary>(Bin.get())) { + if (auto *Universal = dyn_cast<MachOUniversalBinary>(Bin.get())) { // If we have a universal binary, try to look up the object for the // appropriate architecture. auto ObjectFileOrErr = Universal->getObjectForArch(Arch); if (!ObjectFileOrErr) return ObjectFileOrErr.takeError(); OF = std::move(ObjectFileOrErr.get()); - } else if (isa<object::ObjectFile>(Bin.get())) { + } else if (isa<ObjectFile>(Bin.get())) { // For any other object file, upcast and take ownership. - OF.reset(cast<object::ObjectFile>(Bin.release())); + OF.reset(cast<ObjectFile>(Bin.release())); // If we've asked for a particular arch, make sure they match. if (!Arch.empty() && OF->getArch() != Triple(Arch).getArch()) return errorCodeToError(object_error::arch_not_found); @@ -623,11 +651,15 @@ static Error loadBinaryFormat(MemoryBufferRef ObjectBuffer, : support::endianness::big; // Look for the sections that we are interested in. - auto NamesSection = lookupSection(*OF, getInstrProfNameSectionName(false)); + auto ObjFormat = OF->getTripleObjectFormat(); + auto NamesSection = + lookupSection(*OF, getInstrProfSectionName(IPSK_name, ObjFormat, + /*AddSegmentInfo=*/false)); if (auto E = NamesSection.takeError()) return E; auto CoverageSection = - lookupSection(*OF, getInstrProfCoverageSectionName(false)); + lookupSection(*OF, getInstrProfSectionName(IPSK_covmap, ObjFormat, + /*AddSegmentInfo=*/false)); if (auto E = CoverageSection.takeError()) return E; diff --git a/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp b/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp index 8235633..6fe9353 100644 --- a/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp +++ b/contrib/llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp @@ -1,4 +1,4 @@ -//=-- CoverageMappingWriter.cpp - Code coverage mapping writer -------------=// +//===- CoverageMappingWriter.cpp - Code coverage mapping writer -----------===// // // The LLVM Compiler Infrastructure // @@ -13,7 +13,14 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/Coverage/CoverageMappingWriter.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/ADT/SmallVector.h" #include "llvm/Support/LEB128.h" +#include "llvm/Support/raw_ostream.h" +#include <algorithm> +#include <cassert> +#include <limits> +#include <vector> using namespace llvm; using namespace coverage; @@ -27,14 +34,25 @@ void CoverageFilenamesSectionWriter::write(raw_ostream &OS) { } namespace { + /// \brief Gather only the expressions that are used by the mapping /// regions in this function. class CounterExpressionsMinimizer { ArrayRef<CounterExpression> Expressions; - llvm::SmallVector<CounterExpression, 16> UsedExpressions; + SmallVector<CounterExpression, 16> UsedExpressions; std::vector<unsigned> AdjustedExpressionIDs; public: + CounterExpressionsMinimizer(ArrayRef<CounterExpression> Expressions, + ArrayRef<CounterMappingRegion> MappingRegions) + : Expressions(Expressions) { + AdjustedExpressionIDs.resize(Expressions.size(), 0); + for (const auto &I : MappingRegions) + mark(I.Count); + for (const auto &I : MappingRegions) + gatherUsed(I.Count); + } + void mark(Counter C) { if (!C.isExpression()) return; @@ -54,16 +72,6 @@ public: gatherUsed(E.RHS); } - CounterExpressionsMinimizer(ArrayRef<CounterExpression> Expressions, - ArrayRef<CounterMappingRegion> MappingRegions) - : Expressions(Expressions) { - AdjustedExpressionIDs.resize(Expressions.size(), 0); - for (const auto &I : MappingRegions) - mark(I.Count); - for (const auto &I : MappingRegions) - gatherUsed(I.Count); - } - ArrayRef<CounterExpression> getExpressions() const { return UsedExpressions; } /// \brief Adjust the given counter to correctly transition from the old @@ -74,7 +82,8 @@ public: return C; } }; -} + +} // end anonymous namespace /// \brief Encode the counter. /// diff --git a/contrib/llvm/lib/ProfileData/InstrProf.cpp b/contrib/llvm/lib/ProfileData/InstrProf.cpp index 74acd9e..48c1643 100644 --- a/contrib/llvm/lib/ProfileData/InstrProf.cpp +++ b/contrib/llvm/lib/ProfileData/InstrProf.cpp @@ -1,4 +1,4 @@ -//=-- InstrProf.cpp - Instrumented profiling format support -----------------=// +//===- InstrProf.cpp - Instrumented profiling format support --------------===// // // The LLVM Compiler Infrastructure // @@ -13,28 +13,67 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/InstrProf.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/ADT/SmallString.h" +#include "llvm/ADT/SmallVector.h" #include "llvm/ADT/StringExtras.h" +#include "llvm/ADT/StringRef.h" #include "llvm/ADT/Triple.h" +#include "llvm/IR/Constant.h" #include "llvm/IR/Constants.h" #include "llvm/IR/Function.h" +#include "llvm/IR/GlobalValue.h" #include "llvm/IR/GlobalVariable.h" +#include "llvm/IR/Instruction.h" +#include "llvm/IR/LLVMContext.h" #include "llvm/IR/MDBuilder.h" +#include "llvm/IR/Metadata.h" #include "llvm/IR/Module.h" +#include "llvm/IR/Type.h" +#include "llvm/Support/Casting.h" +#include "llvm/Support/CommandLine.h" +#include "llvm/Support/Compiler.h" #include "llvm/Support/Compression.h" +#include "llvm/Support/Endian.h" +#include "llvm/Support/Error.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/LEB128.h" #include "llvm/Support/ManagedStatic.h" +#include "llvm/Support/MathExtras.h" #include "llvm/Support/Path.h" +#include "llvm/Support/SwapByteOrder.h" +#include <algorithm> +#include <cassert> +#include <cstddef> +#include <cstdint> +#include <cstring> +#include <memory> +#include <string> +#include <system_error> +#include <utility> +#include <vector> using namespace llvm; static cl::opt<bool> StaticFuncFullModulePrefix( - "static-func-full-module-prefix", cl::init(false), + "static-func-full-module-prefix", cl::init(true), cl::desc("Use full module build paths in the profile counter names for " "static functions.")); -namespace { -std::string getInstrProfErrString(instrprof_error Err) { +// This option is tailored to users that have different top-level directory in +// profile-gen and profile-use compilation. Users need to specific the number +// of levels to strip. A value larger than the number of directories in the +// source file will strip all the directory names and only leave the basename. +// +// Note current ThinLTO module importing for the indirect-calls assumes +// the source directory name not being stripped. A non-zero option value here +// can potentially prevent some inter-module indirect-call-promotions. +static cl::opt<unsigned> StaticFuncStripDirNamePrefix( + "static-func-strip-dirname-prefix", cl::init(0), + cl::desc("Strip specified level of directory name from source path in " + "the profile counter name for static functions.")); + +static std::string getInstrProfErrString(instrprof_error Err) { switch (Err) { case instrprof_error::success: return "Success"; @@ -76,15 +115,19 @@ std::string getInstrProfErrString(instrprof_error Err) { llvm_unreachable("A value of instrprof_error has no message."); } +namespace { + // FIXME: This class is only here to support the transition to llvm::Error. It // will be removed once this transition is complete. Clients should prefer to // deal with the Error value directly, rather than converting to error_code. class InstrProfErrorCategoryType : public std::error_category { const char *name() const noexcept override { return "llvm.instrprof"; } + std::string message(int IE) const override { return getInstrProfErrString(static_cast<instrprof_error>(IE)); } }; + } // end anonymous namespace static ManagedStatic<InstrProfErrorCategoryType> ErrorCategory; @@ -93,8 +136,49 @@ const std::error_category &llvm::instrprof_category() { return *ErrorCategory; } +namespace { + +const char *InstrProfSectNameCommon[] = { +#define INSTR_PROF_SECT_ENTRY(Kind, SectNameCommon, SectNameCoff, Prefix) \ + SectNameCommon, +#include "llvm/ProfileData/InstrProfData.inc" +}; + +const char *InstrProfSectNameCoff[] = { +#define INSTR_PROF_SECT_ENTRY(Kind, SectNameCommon, SectNameCoff, Prefix) \ + SectNameCoff, +#include "llvm/ProfileData/InstrProfData.inc" +}; + +const char *InstrProfSectNamePrefix[] = { +#define INSTR_PROF_SECT_ENTRY(Kind, SectNameCommon, SectNameCoff, Prefix) \ + Prefix, +#include "llvm/ProfileData/InstrProfData.inc" +}; + +} // namespace + namespace llvm { +std::string getInstrProfSectionName(InstrProfSectKind IPSK, + Triple::ObjectFormatType OF, + bool AddSegmentInfo) { + std::string SectName; + + if (OF == Triple::MachO && AddSegmentInfo) + SectName = InstrProfSectNamePrefix[IPSK]; + + if (OF == Triple::COFF) + SectName += InstrProfSectNameCoff[IPSK]; + else + SectName += InstrProfSectNameCommon[IPSK]; + + if (OF == Triple::MachO && IPSK == IPSK_data && AddSegmentInfo) + SectName += ",regular,live_support"; + + return SectName; +} + void SoftInstrProfErrors::addError(instrprof_error IE) { if (IE == instrprof_error::success) return; @@ -133,6 +217,24 @@ std::string getPGOFuncName(StringRef RawFuncName, return GlobalValue::getGlobalIdentifier(RawFuncName, Linkage, FileName); } +// Strip NumPrefix level of directory name from PathNameStr. If the number of +// directory separators is less than NumPrefix, strip all the directories and +// leave base file name only. +static StringRef stripDirPrefix(StringRef PathNameStr, uint32_t NumPrefix) { + uint32_t Count = NumPrefix; + uint32_t Pos = 0, LastPos = 0; + for (auto & CI : PathNameStr) { + ++Pos; + if (llvm::sys::path::is_separator(CI)) { + LastPos = Pos; + --Count; + } + if (Count == 0) + break; + } + return PathNameStr.substr(LastPos); +} + // Return the PGOFuncName. This function has some special handling when called // in LTO optimization. The following only applies when calling in LTO passes // (when \c InLTO is true): LTO's internalization privatizes many global linkage @@ -151,6 +253,8 @@ std::string getPGOFuncName(const Function &F, bool InLTO, uint64_t Version) { StringRef FileName = (StaticFuncFullModulePrefix ? F.getParent()->getName() : sys::path::filename(F.getParent()->getName())); + if (StaticFuncFullModulePrefix && StaticFuncStripDirNamePrefix != 0) + FileName = stripDirPrefix(FileName, StaticFuncStripDirNamePrefix); return getPGOFuncName(F.getName(), F.getLinkage(), FileName, Version); } @@ -198,7 +302,6 @@ std::string getPGOFuncNameVarName(StringRef FuncName, GlobalVariable *createPGOFuncNameVar(Module &M, GlobalValue::LinkageTypes Linkage, StringRef PGOFuncName) { - // We generally want to match the function's linkage, but available_externally // and extern_weak both have the wrong semantics, and anything that doesn't // need to link across compilation units doesn't need to be visible at all. @@ -227,23 +330,37 @@ GlobalVariable *createPGOFuncNameVar(Function &F, StringRef PGOFuncName) { return createPGOFuncNameVar(*F.getParent(), F.getLinkage(), PGOFuncName); } -void InstrProfSymtab::create(Module &M, bool InLTO) { +Error InstrProfSymtab::create(Module &M, bool InLTO) { for (Function &F : M) { // Function may not have a name: like using asm("") to overwrite the name. // Ignore in this case. if (!F.hasName()) continue; const std::string &PGOFuncName = getPGOFuncName(F, InLTO); - addFuncName(PGOFuncName); + if (Error E = addFuncName(PGOFuncName)) + return E; MD5FuncMap.emplace_back(Function::getGUID(PGOFuncName), &F); + // In ThinLTO, local function may have been promoted to global and have + // suffix added to the function name. We need to add the stripped function + // name to the symbol table so that we can find a match from profile. + if (InLTO) { + auto pos = PGOFuncName.find('.'); + if (pos != std::string::npos) { + const std::string &OtherFuncName = PGOFuncName.substr(0, pos); + if (Error E = addFuncName(OtherFuncName)) + return E; + MD5FuncMap.emplace_back(Function::getGUID(OtherFuncName), &F); + } + } } finalizeSymtab(); + return Error::success(); } -Error collectPGOFuncNameStrings(const std::vector<std::string> &NameStrs, +Error collectPGOFuncNameStrings(ArrayRef<std::string> NameStrs, bool doCompression, std::string &Result) { - assert(NameStrs.size() && "No name data to emit"); + assert(!NameStrs.empty() && "No name data to emit"); uint8_t Header[16], *P = Header; std::string UncompressedNameStrings = @@ -271,12 +388,12 @@ Error collectPGOFuncNameStrings(const std::vector<std::string> &NameStrs, } SmallString<128> CompressedNameStrings; - zlib::Status Success = - zlib::compress(StringRef(UncompressedNameStrings), CompressedNameStrings, - zlib::BestSizeCompression); - - if (Success != zlib::StatusOK) + Error E = zlib::compress(StringRef(UncompressedNameStrings), + CompressedNameStrings, zlib::BestSizeCompression); + if (E) { + consumeError(std::move(E)); return make_error<InstrProfError>(instrprof_error::compress_failed); + } return WriteStringToResult(CompressedNameStrings.size(), CompressedNameStrings); @@ -289,7 +406,7 @@ StringRef getPGOFuncNameVarInitializer(GlobalVariable *NameVar) { return NameStr; } -Error collectPGOFuncNameStrings(const std::vector<GlobalVariable *> &NameVars, +Error collectPGOFuncNameStrings(ArrayRef<GlobalVariable *> NameVars, std::string &Result, bool doCompression) { std::vector<std::string> NameStrs; for (auto *NameVar : NameVars) { @@ -315,9 +432,12 @@ Error readPGOFuncNameStrings(StringRef NameStrings, InstrProfSymtab &Symtab) { if (isCompressed) { StringRef CompressedNameStrings(reinterpret_cast<const char *>(P), CompressedSize); - if (zlib::uncompress(CompressedNameStrings, UncompressedNameStrings, - UncompressedSize) != zlib::StatusOK) + if (Error E = + zlib::uncompress(CompressedNameStrings, UncompressedNameStrings, + UncompressedSize)) { + consumeError(std::move(E)); return make_error<InstrProfError>(instrprof_error::uncompress_failed); + } P += CompressedSize; NameStrings = StringRef(UncompressedNameStrings.data(), UncompressedNameStrings.size()); @@ -330,7 +450,8 @@ Error readPGOFuncNameStrings(StringRef NameStrings, InstrProfSymtab &Symtab) { SmallVector<StringRef, 0> Names; NameStrings.split(Names, getInstrProfNameSeparator()); for (StringRef &Name : Names) - Symtab.addFuncName(Name); + if (Error E = Symtab.addFuncName(Name)) + return E; while (P < EndP && *P == 0) P++; @@ -339,9 +460,9 @@ Error readPGOFuncNameStrings(StringRef NameStrings, InstrProfSymtab &Symtab) { return Error::success(); } -void InstrProfValueSiteRecord::merge(SoftInstrProfErrors &SIPE, - InstrProfValueSiteRecord &Input, - uint64_t Weight) { +void InstrProfValueSiteRecord::merge(InstrProfValueSiteRecord &Input, + uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { this->sortByTargetValues(); Input.sortByTargetValues(); auto I = ValueData.begin(); @@ -354,7 +475,7 @@ void InstrProfValueSiteRecord::merge(SoftInstrProfErrors &SIPE, bool Overflowed; I->Count = SaturatingMultiplyAdd(J->Count, Weight, I->Count, &Overflowed); if (Overflowed) - SIPE.addError(instrprof_error::counter_overflow); + Warn(instrprof_error::counter_overflow); ++I; continue; } @@ -362,40 +483,43 @@ void InstrProfValueSiteRecord::merge(SoftInstrProfErrors &SIPE, } } -void InstrProfValueSiteRecord::scale(SoftInstrProfErrors &SIPE, - uint64_t Weight) { +void InstrProfValueSiteRecord::scale(uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { for (auto I = ValueData.begin(), IE = ValueData.end(); I != IE; ++I) { bool Overflowed; I->Count = SaturatingMultiply(I->Count, Weight, &Overflowed); if (Overflowed) - SIPE.addError(instrprof_error::counter_overflow); + Warn(instrprof_error::counter_overflow); } } // Merge Value Profile data from Src record to this record for ValueKind. // Scale merged value counts by \p Weight. -void InstrProfRecord::mergeValueProfData(uint32_t ValueKind, - InstrProfRecord &Src, - uint64_t Weight) { +void InstrProfRecord::mergeValueProfData( + uint32_t ValueKind, InstrProfRecord &Src, uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { uint32_t ThisNumValueSites = getNumValueSites(ValueKind); uint32_t OtherNumValueSites = Src.getNumValueSites(ValueKind); if (ThisNumValueSites != OtherNumValueSites) { - SIPE.addError(instrprof_error::value_site_count_mismatch); + Warn(instrprof_error::value_site_count_mismatch); return; } + if (!ThisNumValueSites) + return; std::vector<InstrProfValueSiteRecord> &ThisSiteRecords = - getValueSitesForKind(ValueKind); - std::vector<InstrProfValueSiteRecord> &OtherSiteRecords = + getOrCreateValueSitesForKind(ValueKind); + MutableArrayRef<InstrProfValueSiteRecord> OtherSiteRecords = Src.getValueSitesForKind(ValueKind); for (uint32_t I = 0; I < ThisNumValueSites; I++) - ThisSiteRecords[I].merge(SIPE, OtherSiteRecords[I], Weight); + ThisSiteRecords[I].merge(OtherSiteRecords[I], Weight, Warn); } -void InstrProfRecord::merge(InstrProfRecord &Other, uint64_t Weight) { +void InstrProfRecord::merge(InstrProfRecord &Other, uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { // If the number of counters doesn't match we either have bad data // or a hash collision. if (Counts.size() != Other.Counts.size()) { - SIPE.addError(instrprof_error::count_mismatch); + Warn(instrprof_error::count_mismatch); return; } @@ -404,30 +528,30 @@ void InstrProfRecord::merge(InstrProfRecord &Other, uint64_t Weight) { Counts[I] = SaturatingMultiplyAdd(Other.Counts[I], Weight, Counts[I], &Overflowed); if (Overflowed) - SIPE.addError(instrprof_error::counter_overflow); + Warn(instrprof_error::counter_overflow); } for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind) - mergeValueProfData(Kind, Other, Weight); + mergeValueProfData(Kind, Other, Weight, Warn); } -void InstrProfRecord::scaleValueProfData(uint32_t ValueKind, uint64_t Weight) { - uint32_t ThisNumValueSites = getNumValueSites(ValueKind); - std::vector<InstrProfValueSiteRecord> &ThisSiteRecords = - getValueSitesForKind(ValueKind); - for (uint32_t I = 0; I < ThisNumValueSites; I++) - ThisSiteRecords[I].scale(SIPE, Weight); +void InstrProfRecord::scaleValueProfData( + uint32_t ValueKind, uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { + for (auto &R : getValueSitesForKind(ValueKind)) + R.scale(Weight, Warn); } -void InstrProfRecord::scale(uint64_t Weight) { +void InstrProfRecord::scale(uint64_t Weight, + function_ref<void(instrprof_error)> Warn) { for (auto &Count : this->Counts) { bool Overflowed; Count = SaturatingMultiply(Count, Weight, &Overflowed); if (Overflowed) - SIPE.addError(instrprof_error::counter_overflow); + Warn(instrprof_error::counter_overflow); } for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind) - scaleValueProfData(Kind, Weight); + scaleValueProfData(Kind, Weight, Warn); } // Map indirect call target name hash to name string. @@ -462,7 +586,7 @@ void InstrProfRecord::addValueData(uint32_t ValueKind, uint32_t Site, VData[I].Value = remapValue(VData[I].Value, ValueKind, ValueMap); } std::vector<InstrProfValueSiteRecord> &ValueSites = - getValueSitesForKind(ValueKind); + getOrCreateValueSitesForKind(ValueKind); if (N == 0) ValueSites.emplace_back(); else @@ -521,8 +645,9 @@ static ValueProfRecordClosure InstrProfRecordClosure = { // Wrapper implementation using the closure mechanism. uint32_t ValueProfData::getSize(const InstrProfRecord &Record) { - InstrProfRecordClosure.Record = &Record; - return getValueProfDataSize(&InstrProfRecordClosure); + auto Closure = InstrProfRecordClosure; + Closure.Record = &Record; + return getValueProfDataSize(&Closure); } // Wrapper implementation using the closure mechanism. @@ -553,6 +678,7 @@ void ValueProfRecord::deserializeTo(InstrProfRecord &Record, void ValueProfRecord::swapBytes(support::endianness Old, support::endianness New) { using namespace support; + if (Old == New) return; @@ -589,6 +715,7 @@ void ValueProfData::deserializeTo(InstrProfRecord &Record, template <class T> static T swapToHostOrder(const unsigned char *&D, support::endianness Orig) { using namespace support; + if (Orig == little) return endian::readNext<T, little, unaligned>(D); else @@ -623,6 +750,7 @@ ValueProfData::getValueProfData(const unsigned char *D, const unsigned char *const BufferEnd, support::endianness Endianness) { using namespace support; + if (D + sizeof(ValueProfData) > BufferEnd) return make_error<InstrProfError>(instrprof_error::truncated); @@ -645,6 +773,7 @@ ValueProfData::getValueProfData(const unsigned char *D, void ValueProfData::swapBytesToHost(support::endianness Endianness) { using namespace support; + if (Endianness == getHostEndianness()) return; @@ -660,6 +789,7 @@ void ValueProfData::swapBytesToHost(support::endianness Endianness) { void ValueProfData::swapBytesFromHost(support::endianness Endianness) { using namespace support; + if (Endianness == getHostEndianness()) return; @@ -791,7 +921,7 @@ bool needsComdatForCounter(const Function &F, const Module &M) { return true; Triple TT(M.getTargetTriple()); - if (!TT.isOSBinFormatELF()) + if (!TT.isOSBinFormatELF() && !TT.isOSBinFormatWasm()) return false; // See createPGOFuncNameVar for more details. To avoid link errors, profile @@ -854,4 +984,26 @@ bool canRenameComdatFunc(const Function &F, bool CheckAddressTaken) { } return true; } + +// Parse the value profile options. +void getMemOPSizeRangeFromOption(StringRef MemOPSizeRange, int64_t &RangeStart, + int64_t &RangeLast) { + static const int64_t DefaultMemOPSizeRangeStart = 0; + static const int64_t DefaultMemOPSizeRangeLast = 8; + RangeStart = DefaultMemOPSizeRangeStart; + RangeLast = DefaultMemOPSizeRangeLast; + + if (!MemOPSizeRange.empty()) { + auto Pos = MemOPSizeRange.find(':'); + if (Pos != std::string::npos) { + if (Pos > 0) + MemOPSizeRange.substr(0, Pos).getAsInteger(10, RangeStart); + if (Pos < MemOPSizeRange.size() - 1) + MemOPSizeRange.substr(Pos + 1).getAsInteger(10, RangeLast); + } else + MemOPSizeRange.getAsInteger(10, RangeLast); + } + assert(RangeLast >= RangeStart); +} + } // end namespace llvm diff --git a/contrib/llvm/lib/ProfileData/InstrProfReader.cpp b/contrib/llvm/lib/ProfileData/InstrProfReader.cpp index ad407f0..1b39a06 100644 --- a/contrib/llvm/lib/ProfileData/InstrProfReader.cpp +++ b/contrib/llvm/lib/ProfileData/InstrProfReader.cpp @@ -1,4 +1,4 @@ -//=-- InstrProfReader.cpp - Instrumented profiling reader -------------------=// +//===- InstrProfReader.cpp - Instrumented profiling reader ----------------===// // // The LLVM Compiler Infrastructure // @@ -13,8 +13,26 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/InstrProfReader.h" +#include "llvm/ADT/ArrayRef.h" #include "llvm/ADT/STLExtras.h" -#include <cassert> +#include "llvm/ADT/StringRef.h" +#include "llvm/IR/ProfileSummary.h" +#include "llvm/ProfileData/InstrProf.h" +#include "llvm/ProfileData/ProfileCommon.h" +#include "llvm/Support/Endian.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/ErrorOr.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/SwapByteOrder.h" +#include <algorithm> +#include <cctype> +#include <cstddef> +#include <cstdint> +#include <limits> +#include <memory> +#include <system_error> +#include <utility> +#include <vector> using namespace llvm; @@ -78,7 +96,6 @@ IndexedInstrProfReader::create(const Twine &Path) { return IndexedInstrProfReader::create(std::move(BufferOrError.get())); } - Expected<std::unique_ptr<IndexedInstrProfReader>> IndexedInstrProfReader::create(std::unique_ptr<MemoryBuffer> Buffer) { // Sanity check the buffer. @@ -182,8 +199,9 @@ TextInstrProfReader::readValueProfileData(InstrProfRecord &Record) { CHECK_LINE_END(Line); std::pair<StringRef, StringRef> VD = Line->rsplit(':'); uint64_t TakenCount, Value; - if (VK == IPVK_IndirectCallTarget) { - Symtab->addFuncName(VD.first); + if (ValueKind == IPVK_IndirectCallTarget) { + if (Error E = Symtab->addFuncName(VD.first)) + return E; Value = IndexedInstrProf::ComputeHash(VD.first); } else { READ_NUM(VD.first, Value); @@ -192,7 +210,8 @@ TextInstrProfReader::readValueProfileData(InstrProfRecord &Record) { CurrentValues.push_back({Value, TakenCount}); Line++; } - Record.addValueData(VK, S, CurrentValues.data(), NumValueData, nullptr); + Record.addValueData(ValueKind, S, CurrentValues.data(), NumValueData, + nullptr); } } return success(); @@ -202,7 +221,7 @@ TextInstrProfReader::readValueProfileData(InstrProfRecord &Record) { #undef VP_READ_ADVANCE } -Error TextInstrProfReader::readNextRecord(InstrProfRecord &Record) { +Error TextInstrProfReader::readNextRecord(NamedInstrProfRecord &Record) { // Skip empty lines and comments. while (!Line.is_at_end() && (Line->empty() || Line->startswith("#"))) ++Line; @@ -214,7 +233,8 @@ Error TextInstrProfReader::readNextRecord(InstrProfRecord &Record) { // Read the function name. Record.Name = *Line++; - Symtab->addFuncName(Record.Name); + if (Error E = Symtab->addFuncName(Record.Name)) + return E; // Read the function hash. if (Line.is_at_end()) @@ -232,7 +252,7 @@ Error TextInstrProfReader::readNextRecord(InstrProfRecord &Record) { return error(instrprof_error::malformed); // Read each counter and fill our internal storage with the values. - Record.Counts.clear(); + Record.Clear(); Record.Counts.reserve(NumCounters); for (uint64_t I = 0; I < NumCounters; ++I) { if (Line.is_at_end()) @@ -357,13 +377,13 @@ Error RawInstrProfReader<IntPtrT>::readHeader( } template <class IntPtrT> -Error RawInstrProfReader<IntPtrT>::readName(InstrProfRecord &Record) { +Error RawInstrProfReader<IntPtrT>::readName(NamedInstrProfRecord &Record) { Record.Name = getName(Data->NameRef); return success(); } template <class IntPtrT> -Error RawInstrProfReader<IntPtrT>::readFuncHash(InstrProfRecord &Record) { +Error RawInstrProfReader<IntPtrT>::readFuncHash(NamedInstrProfRecord &Record) { Record.Hash = swap(Data->FuncHash); return success(); } @@ -398,7 +418,6 @@ Error RawInstrProfReader<IntPtrT>::readRawCounts( template <class IntPtrT> Error RawInstrProfReader<IntPtrT>::readValueProfilingData( InstrProfRecord &Record) { - Record.clearValueData(); CurValueDataSize = 0; // Need to match the logic in value profile dumper code in compiler-rt: @@ -426,7 +445,7 @@ Error RawInstrProfReader<IntPtrT>::readValueProfilingData( } template <class IntPtrT> -Error RawInstrProfReader<IntPtrT>::readNextRecord(InstrProfRecord &Record) { +Error RawInstrProfReader<IntPtrT>::readNextRecord(NamedInstrProfRecord &Record) { if (atEnd()) // At this point, ValueDataStart field points to the next header. if (Error E = readNextHeader(getNextHeaderPos())) @@ -454,17 +473,19 @@ Error RawInstrProfReader<IntPtrT>::readNextRecord(InstrProfRecord &Record) { } namespace llvm { + template class RawInstrProfReader<uint32_t>; template class RawInstrProfReader<uint64_t>; -} + +} // end namespace llvm InstrProfLookupTrait::hash_value_type InstrProfLookupTrait::ComputeHash(StringRef K) { return IndexedInstrProf::ComputeHash(HashType, K); } -typedef InstrProfLookupTrait::data_type data_type; -typedef InstrProfLookupTrait::offset_type offset_type; +using data_type = InstrProfLookupTrait::data_type; +using offset_type = InstrProfLookupTrait::offset_type; bool InstrProfLookupTrait::readValueProfilingData( const unsigned char *&D, const unsigned char *const End) { @@ -482,6 +503,8 @@ bool InstrProfLookupTrait::readValueProfilingData( data_type InstrProfLookupTrait::ReadData(StringRef K, const unsigned char *D, offset_type N) { + using namespace support; + // Check if the data is corrupt. If so, don't try to read it. if (N % sizeof(uint64_t)) return data_type(); @@ -489,7 +512,6 @@ data_type InstrProfLookupTrait::ReadData(StringRef K, const unsigned char *D, DataBuffer.clear(); std::vector<uint64_t> CounterBuffer; - using namespace support; const unsigned char *End = D + N; while (D < End) { // Read hash. @@ -528,7 +550,7 @@ data_type InstrProfLookupTrait::ReadData(StringRef K, const unsigned char *D, template <typename HashTableImpl> Error InstrProfReaderIndex<HashTableImpl>::getRecords( - StringRef FuncName, ArrayRef<InstrProfRecord> &Data) { + StringRef FuncName, ArrayRef<NamedInstrProfRecord> &Data) { auto Iter = HashTable->find(FuncName); if (Iter == HashTable->end()) return make_error<InstrProfError>(instrprof_error::unknown_function); @@ -542,7 +564,7 @@ Error InstrProfReaderIndex<HashTableImpl>::getRecords( template <typename HashTableImpl> Error InstrProfReaderIndex<HashTableImpl>::getRecords( - ArrayRef<InstrProfRecord> &Data) { + ArrayRef<NamedInstrProfRecord> &Data) { if (atEnd()) return make_error<InstrProfError>(instrprof_error::eof); @@ -567,9 +589,10 @@ InstrProfReaderIndex<HashTableImpl>::InstrProfReaderIndex( } bool IndexedInstrProfReader::hasFormat(const MemoryBuffer &DataBuffer) { + using namespace support; + if (DataBuffer.getBufferSize() < 8) return false; - using namespace support; uint64_t Magic = endian::read<uint64_t, little, aligned>(DataBuffer.getBufferStart()); // Verify that it's magical. @@ -581,6 +604,7 @@ IndexedInstrProfReader::readSummary(IndexedInstrProf::ProfVersion Version, const unsigned char *Cur) { using namespace IndexedInstrProf; using namespace support; + if (Version >= IndexedInstrProf::Version4) { const IndexedInstrProf::Summary *SummaryInLE = reinterpret_cast<const IndexedInstrProf::Summary *>(Cur); @@ -598,7 +622,7 @@ IndexedInstrProfReader::readSummary(IndexedInstrProf::ProfVersion Version, for (unsigned I = 0; I < SummarySize / sizeof(uint64_t); I++) Dst[I] = endian::byte_swap<uint64_t, little>(Src[I]); - llvm::SummaryEntryVector DetailedSummary; + SummaryEntryVector DetailedSummary; for (unsigned I = 0; I < SummaryData->NumCutoffEntries; I++) { const IndexedInstrProf::Summary::Entry &Ent = SummaryData->getEntry(I); DetailedSummary.emplace_back((uint32_t)Ent.Cutoff, Ent.MinBlockCount, @@ -617,23 +641,24 @@ IndexedInstrProfReader::readSummary(IndexedInstrProf::ProfVersion Version, } else { // For older version of profile data, we need to compute on the fly: using namespace IndexedInstrProf; + InstrProfSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs); // FIXME: This only computes an empty summary. Need to call addRecord for - // all InstrProfRecords to get the correct summary. + // all NamedInstrProfRecords to get the correct summary. this->Summary = Builder.getSummary(); return Cur; } } Error IndexedInstrProfReader::readHeader() { + using namespace support; + const unsigned char *Start = (const unsigned char *)DataBuffer->getBufferStart(); const unsigned char *Cur = Start; if ((const unsigned char *)DataBuffer->getBufferEnd() - Cur < 24) return error(instrprof_error::truncated); - using namespace support; - auto *Header = reinterpret_cast<const IndexedInstrProf::Header *>(Cur); Cur += sizeof(IndexedInstrProf::Header); @@ -671,7 +696,9 @@ InstrProfSymtab &IndexedInstrProfReader::getSymtab() { return *Symtab.get(); std::unique_ptr<InstrProfSymtab> NewSymtab = make_unique<InstrProfSymtab>(); - Index->populateSymtab(*NewSymtab.get()); + if (Error E = Index->populateSymtab(*NewSymtab.get())) { + consumeError(error(InstrProfError::take(std::move(E)))); + } Symtab = std::move(NewSymtab); return *Symtab.get(); @@ -680,7 +707,7 @@ InstrProfSymtab &IndexedInstrProfReader::getSymtab() { Expected<InstrProfRecord> IndexedInstrProfReader::getInstrProfRecord(StringRef FuncName, uint64_t FuncHash) { - ArrayRef<InstrProfRecord> Data; + ArrayRef<NamedInstrProfRecord> Data; Error Err = Index->getRecords(FuncName, Data); if (Err) return std::move(Err); @@ -705,10 +732,10 @@ Error IndexedInstrProfReader::getFunctionCounts(StringRef FuncName, return success(); } -Error IndexedInstrProfReader::readNextRecord(InstrProfRecord &Record) { +Error IndexedInstrProfReader::readNextRecord(NamedInstrProfRecord &Record) { static unsigned RecordIndex = 0; - ArrayRef<InstrProfRecord> Data; + ArrayRef<NamedInstrProfRecord> Data; Error E = Index->getRecords(Data); if (E) diff --git a/contrib/llvm/lib/ProfileData/InstrProfWriter.cpp b/contrib/llvm/lib/ProfileData/InstrProfWriter.cpp index 029d756..ce3f880 100644 --- a/contrib/llvm/lib/ProfileData/InstrProfWriter.cpp +++ b/contrib/llvm/lib/ProfileData/InstrProfWriter.cpp @@ -1,4 +1,4 @@ -//=-- InstrProfWriter.cpp - Instrumented profiling writer -------------------=// +//===- InstrProfWriter.cpp - Instrumented profiling writer ----------------===// // // The LLVM Compiler Infrastructure // @@ -13,14 +13,20 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/InstrProfWriter.h" +#include "llvm/ADT/STLExtras.h" #include "llvm/ADT/StringRef.h" #include "llvm/IR/ProfileSummary.h" +#include "llvm/ProfileData/InstrProf.h" #include "llvm/ProfileData/ProfileCommon.h" +#include "llvm/Support/Endian.h" #include "llvm/Support/EndianStream.h" +#include "llvm/Support/Error.h" #include "llvm/Support/MemoryBuffer.h" #include "llvm/Support/OnDiskHashTable.h" #include "llvm/Support/raw_ostream.h" #include <algorithm> +#include <cstdint> +#include <memory> #include <string> #include <tuple> #include <utility> @@ -41,10 +47,9 @@ namespace llvm { // A wrapper class to abstract writer stream with support of bytes // back patching. class ProfOStream { - public: - ProfOStream(llvm::raw_fd_ostream &FD) : IsFDOStream(true), OS(FD), LE(FD) {} - ProfOStream(llvm::raw_string_ostream &STR) + ProfOStream(raw_fd_ostream &FD) : IsFDOStream(true), OS(FD), LE(FD) {} + ProfOStream(raw_string_ostream &STR) : IsFDOStream(false), OS(STR), LE(STR) {} uint64_t tell() { return OS.tell(); } @@ -55,16 +60,16 @@ public: // directly and it won't be reflected in the stream's internal buffer. void patch(PatchItem *P, int NItems) { using namespace support; + if (IsFDOStream) { - llvm::raw_fd_ostream &FDOStream = static_cast<llvm::raw_fd_ostream &>(OS); + raw_fd_ostream &FDOStream = static_cast<raw_fd_ostream &>(OS); for (int K = 0; K < NItems; K++) { FDOStream.seek(P[K].Pos); for (int I = 0; I < P[K].N; I++) write(P[K].D[I]); } } else { - llvm::raw_string_ostream &SOStream = - static_cast<llvm::raw_string_ostream &>(OS); + raw_string_ostream &SOStream = static_cast<raw_string_ostream &>(OS); std::string &Data = SOStream.str(); // with flush for (int K = 0; K < NItems; K++) { for (int I = 0; I < P[K].N; I++) { @@ -85,26 +90,28 @@ public: class InstrProfRecordWriterTrait { public: - typedef StringRef key_type; - typedef StringRef key_type_ref; + using key_type = StringRef; + using key_type_ref = StringRef; - typedef const InstrProfWriter::ProfilingData *const data_type; - typedef const InstrProfWriter::ProfilingData *const data_type_ref; + using data_type = const InstrProfWriter::ProfilingData *const; + using data_type_ref = const InstrProfWriter::ProfilingData *const; - typedef uint64_t hash_value_type; - typedef uint64_t offset_type; + using hash_value_type = uint64_t; + using offset_type = uint64_t; - support::endianness ValueProfDataEndianness; + support::endianness ValueProfDataEndianness = support::little; InstrProfSummaryBuilder *SummaryBuilder; - InstrProfRecordWriterTrait() : ValueProfDataEndianness(support::little) {} + InstrProfRecordWriterTrait() = default; + static hash_value_type ComputeHash(key_type_ref K) { return IndexedInstrProf::ComputeHash(K); } static std::pair<offset_type, offset_type> EmitKeyDataLength(raw_ostream &Out, key_type_ref K, data_type_ref V) { - using namespace llvm::support; + using namespace support; + endian::Writer<little> LE(Out); offset_type N = K.size(); @@ -130,7 +137,8 @@ public: } void EmitData(raw_ostream &Out, key_type_ref, data_type_ref V, offset_type) { - using namespace llvm::support; + using namespace support; + endian::Writer<little> LE(Out); for (const auto &ProfileData : *V) { const InstrProfRecord &ProfRecord = ProfileData.second; @@ -154,8 +162,7 @@ public: } // end namespace llvm InstrProfWriter::InstrProfWriter(bool Sparse) - : Sparse(Sparse), FunctionData(), ProfileKind(PF_Unknown), - InfoObj(new InstrProfRecordWriterTrait()) {} + : Sparse(Sparse), InfoObj(new InstrProfRecordWriterTrait()) {} InstrProfWriter::~InstrProfWriter() { delete InfoObj; } @@ -169,38 +176,46 @@ void InstrProfWriter::setOutputSparse(bool Sparse) { this->Sparse = Sparse; } -Error InstrProfWriter::addRecord(InstrProfRecord &&I, uint64_t Weight) { - auto &ProfileDataMap = FunctionData[I.Name]; +void InstrProfWriter::addRecord(NamedInstrProfRecord &&I, uint64_t Weight, + function_ref<void(Error)> Warn) { + auto Name = I.Name; + auto Hash = I.Hash; + addRecord(Name, Hash, std::move(I), Weight, Warn); +} + +void InstrProfWriter::addRecord(StringRef Name, uint64_t Hash, + InstrProfRecord &&I, uint64_t Weight, + function_ref<void(Error)> Warn) { + auto &ProfileDataMap = FunctionData[Name]; bool NewFunc; ProfilingData::iterator Where; std::tie(Where, NewFunc) = - ProfileDataMap.insert(std::make_pair(I.Hash, InstrProfRecord())); + ProfileDataMap.insert(std::make_pair(Hash, InstrProfRecord())); InstrProfRecord &Dest = Where->second; + auto MapWarn = [&](instrprof_error E) { + Warn(make_error<InstrProfError>(E)); + }; + if (NewFunc) { // We've never seen a function with this name and hash, add it. Dest = std::move(I); - // Fix up the name to avoid dangling reference. - Dest.Name = FunctionData.find(Dest.Name)->getKey(); if (Weight > 1) - Dest.scale(Weight); + Dest.scale(Weight, MapWarn); } else { // We're updating a function we've seen before. - Dest.merge(I, Weight); + Dest.merge(I, Weight, MapWarn); } Dest.sortValueData(); - - return Dest.takeError(); } -Error InstrProfWriter::mergeRecordsFromWriter(InstrProfWriter &&IPW) { +void InstrProfWriter::mergeRecordsFromWriter(InstrProfWriter &&IPW, + function_ref<void(Error)> Warn) { for (auto &I : IPW.FunctionData) for (auto &Func : I.getValue()) - if (Error E = addRecord(std::move(Func.second), 1)) - return E; - return Error::success(); + addRecord(I.getKey(), Func.first, std::move(Func.second), 1, Warn); } bool InstrProfWriter::shouldEncodeData(const ProfilingData &PD) { @@ -208,7 +223,7 @@ bool InstrProfWriter::shouldEncodeData(const ProfilingData &PD) { return true; for (const auto &Func : PD) { const InstrProfRecord &IPR = Func.second; - if (any_of(IPR.Counts, [](uint64_t Count) { return Count > 0; })) + if (llvm::any_of(IPR.Counts, [](uint64_t Count) { return Count > 0; })) return true; } return false; @@ -217,6 +232,7 @@ bool InstrProfWriter::shouldEncodeData(const ProfilingData &PD) { static void setSummary(IndexedInstrProf::Summary *TheSummary, ProfileSummary &PS) { using namespace IndexedInstrProf; + std::vector<ProfileSummaryEntry> &Res = PS.getDetailedSummary(); TheSummary->NumSummaryFields = Summary::NumKinds; TheSummary->NumCutoffEntries = Res.size(); @@ -231,9 +247,10 @@ static void setSummary(IndexedInstrProf::Summary *TheSummary, } void InstrProfWriter::writeImpl(ProfOStream &OS) { + using namespace IndexedInstrProf; + OnDiskChainedHashTableGenerator<InstrProfRecordWriterTrait> Generator; - using namespace IndexedInstrProf; InstrProfSummaryBuilder ISB(ProfileSummaryBuilder::DefaultCutoffs); InfoObj->SummaryBuilder = &ISB; @@ -301,7 +318,7 @@ void InstrProfWriter::write(raw_fd_ostream &OS) { std::unique_ptr<MemoryBuffer> InstrProfWriter::writeBuffer() { std::string Data; - llvm::raw_string_ostream OS(Data); + raw_string_ostream OS(Data); ProfOStream POS(OS); // Write the hash table. writeImpl(POS); @@ -314,11 +331,12 @@ static const char *ValueProfKindStr[] = { #include "llvm/ProfileData/InstrProfData.inc" }; -void InstrProfWriter::writeRecordInText(const InstrProfRecord &Func, +void InstrProfWriter::writeRecordInText(StringRef Name, uint64_t Hash, + const InstrProfRecord &Func, InstrProfSymtab &Symtab, raw_fd_ostream &OS) { - OS << Func.Name << "\n"; - OS << "# Func Hash:\n" << Func.Hash << "\n"; + OS << Name << "\n"; + OS << "# Func Hash:\n" << Hash << "\n"; OS << "# Num Counters:\n" << Func.Counts.size() << "\n"; OS << "# Counter Values:\n"; for (uint64_t Count : Func.Counts) @@ -353,17 +371,19 @@ void InstrProfWriter::writeRecordInText(const InstrProfRecord &Func, OS << "\n"; } -void InstrProfWriter::writeText(raw_fd_ostream &OS) { +Error InstrProfWriter::writeText(raw_fd_ostream &OS) { if (ProfileKind == PF_IRLevel) OS << "# IR level Instrumentation Flag\n:ir\n"; InstrProfSymtab Symtab; for (const auto &I : FunctionData) if (shouldEncodeData(I.getValue())) - Symtab.addFuncName(I.getKey()); + if (Error E = Symtab.addFuncName(I.getKey())) + return E; Symtab.finalizeSymtab(); for (const auto &I : FunctionData) if (shouldEncodeData(I.getValue())) for (const auto &Func : I.getValue()) - writeRecordInText(Func.second, Symtab, OS); + writeRecordInText(I.getKey(), Func.first, Func.second, Symtab, OS); + return Error::success(); } diff --git a/contrib/llvm/lib/ProfileData/SampleProf.cpp b/contrib/llvm/lib/ProfileData/SampleProf.cpp index 5bcfff0..eafdd21 100644 --- a/contrib/llvm/lib/ProfileData/SampleProf.cpp +++ b/contrib/llvm/lib/ProfileData/SampleProf.cpp @@ -13,18 +13,25 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/SampleProf.h" +#include "llvm/Support/Compiler.h" +#include "llvm/Support/Debug.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ManagedStatic.h" +#include "llvm/Support/raw_ostream.h" +#include <string> +#include <system_error> -using namespace llvm::sampleprof; using namespace llvm; +using namespace sampleprof; namespace { + // FIXME: This class is only here to support the transition to llvm::Error. It // will be removed once this transition is complete. Clients should prefer to // deal with the Error value directly, rather than converting to error_code. class SampleProfErrorCategoryType : public std::error_category { const char *name() const noexcept override { return "llvm.sampleprof"; } + std::string message(int IE) const override { sampleprof_error E = static_cast<sampleprof_error>(IE); switch (E) { @@ -54,7 +61,8 @@ class SampleProfErrorCategoryType : public std::error_category { llvm_unreachable("A value of sampleprof_error has no message."); } }; -} + +} // end anonymous namespace static ManagedStatic<SampleProfErrorCategoryType> ErrorCategory; @@ -74,7 +82,9 @@ raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS, return OS; } +#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP) LLVM_DUMP_METHOD void LineLocation::dump() const { print(dbgs()); } +#endif /// \brief Print the sample record to the stream \p OS indented by \p Indent. void SampleRecord::print(raw_ostream &OS, unsigned Indent) const { @@ -87,7 +97,9 @@ void SampleRecord::print(raw_ostream &OS, unsigned Indent) const { OS << "\n"; } +#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP) LLVM_DUMP_METHOD void SampleRecord::dump() const { print(dbgs(), 0); } +#endif raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS, const SampleRecord &Sample) { @@ -101,7 +113,7 @@ void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const { << " sampled lines\n"; OS.indent(Indent); - if (BodySamples.size() > 0) { + if (!BodySamples.empty()) { OS << "Samples collected in the function's body {\n"; SampleSorter<LineLocation, SampleRecord> SortedBodySamples(BodySamples); for (const auto &SI : SortedBodySamples.get()) { @@ -115,14 +127,16 @@ void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const { } OS.indent(Indent); - if (CallsiteSamples.size() > 0) { + if (!CallsiteSamples.empty()) { OS << "Samples collected in inlined callsites {\n"; - SampleSorter<LineLocation, FunctionSamples> SortedCallsiteSamples( + SampleSorter<LineLocation, FunctionSamplesMap> SortedCallsiteSamples( CallsiteSamples); for (const auto &CS : SortedCallsiteSamples.get()) { - OS.indent(Indent + 2); - OS << CS->first << ": inlined callee: " << CS->second.getName() << ": "; - CS->second.print(OS, Indent + 4); + for (const auto &FS : CS->second) { + OS.indent(Indent + 2); + OS << CS->first << ": inlined callee: " << FS.second.getName() << ": "; + FS.second.print(OS, Indent + 4); + } } OS << "}\n"; } else { @@ -136,4 +150,6 @@ raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS, return OS; } -void FunctionSamples::dump(void) const { print(dbgs(), 0); } +#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP) +LLVM_DUMP_METHOD void FunctionSamples::dump() const { print(dbgs(), 0); } +#endif diff --git a/contrib/llvm/lib/ProfileData/SampleProfReader.cpp b/contrib/llvm/lib/ProfileData/SampleProfReader.cpp index af80b03..234fe02 100644 --- a/contrib/llvm/lib/ProfileData/SampleProfReader.cpp +++ b/contrib/llvm/lib/ProfileData/SampleProfReader.cpp @@ -23,14 +23,25 @@ #include "llvm/ProfileData/SampleProfReader.h" #include "llvm/ADT/DenseMap.h" #include "llvm/ADT/STLExtras.h" -#include "llvm/Support/Debug.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/IR/ProfileSummary.h" +#include "llvm/ProfileData/ProfileCommon.h" +#include "llvm/ProfileData/SampleProf.h" #include "llvm/Support/ErrorOr.h" #include "llvm/Support/LEB128.h" #include "llvm/Support/LineIterator.h" #include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/raw_ostream.h" +#include <algorithm> +#include <cstddef> +#include <cstdint> +#include <limits> +#include <memory> +#include <system_error> +#include <vector> -using namespace llvm::sampleprof; using namespace llvm; +using namespace sampleprof; /// \brief Dump the function profile for \p FName. /// @@ -200,7 +211,7 @@ std::error_code SampleProfileReaderText::read() { InlineStack.pop_back(); } FunctionSamples &FSamples = InlineStack.back()->functionSamplesAt( - LineLocation(LineOffset, Discriminator)); + LineLocation(LineOffset, Discriminator))[FName]; FSamples.setName(FName); MergeResult(Result, FSamples.addTotalSamples(NumSamples)); InlineStack.push_back(&FSamples); @@ -352,8 +363,8 @@ SampleProfileReaderBinary::readProfile(FunctionSamples &FProfile) { if (std::error_code EC = FName.getError()) return EC; - FunctionSamples &CalleeProfile = - FProfile.functionSamplesAt(LineLocation(*LineOffset, *Discriminator)); + FunctionSamples &CalleeProfile = FProfile.functionSamplesAt( + LineLocation(*LineOffset, *Discriminator))[*FName]; CalleeProfile.setName(*FName); if (std::error_code EC = readProfile(CalleeProfile)) return EC; @@ -625,7 +636,7 @@ std::error_code SampleProfileReaderGCC::readOneFunctionProfile( uint32_t LineOffset = Offset >> 16; uint32_t Discriminator = Offset & 0xffff; FProfile = &CallerProfile->functionSamplesAt( - LineLocation(LineOffset, Discriminator)); + LineLocation(LineOffset, Discriminator))[Name]; } FProfile->setName(Name); @@ -681,11 +692,9 @@ std::error_code SampleProfileReaderGCC::readOneFunctionProfile( if (!GcovBuffer.readInt64(TargetCount)) return sampleprof_error::truncated; - if (Update) { - FunctionSamples &TargetProfile = Profiles[TargetName]; - TargetProfile.addCalledTargetSamples(LineOffset, Discriminator, - TargetName, TargetCount); - } + if (Update) + FProfile->addCalledTargetSamples(LineOffset, Discriminator, + TargetName, TargetCount); } } diff --git a/contrib/llvm/lib/ProfileData/SampleProfWriter.cpp b/contrib/llvm/lib/ProfileData/SampleProfWriter.cpp index 4fa7128..b450261 100644 --- a/contrib/llvm/lib/ProfileData/SampleProfWriter.cpp +++ b/contrib/llvm/lib/ProfileData/SampleProfWriter.cpp @@ -19,15 +19,49 @@ //===----------------------------------------------------------------------===// #include "llvm/ProfileData/SampleProfWriter.h" -#include "llvm/Support/Debug.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/ProfileData/ProfileCommon.h" +#include "llvm/ProfileData/SampleProf.h" #include "llvm/Support/ErrorOr.h" +#include "llvm/Support/FileSystem.h" #include "llvm/Support/LEB128.h" -#include "llvm/Support/LineIterator.h" -#include "llvm/Support/MemoryBuffer.h" -#include "llvm/Support/Regex.h" +#include "llvm/Support/raw_ostream.h" +#include <algorithm> +#include <cstdint> +#include <memory> +#include <set> +#include <system_error> +#include <utility> +#include <vector> -using namespace llvm::sampleprof; using namespace llvm; +using namespace sampleprof; + +std::error_code +SampleProfileWriter::write(const StringMap<FunctionSamples> &ProfileMap) { + if (std::error_code EC = writeHeader(ProfileMap)) + return EC; + + // Sort the ProfileMap by total samples. + typedef std::pair<StringRef, const FunctionSamples *> NameFunctionSamples; + std::vector<NameFunctionSamples> V; + for (const auto &I : ProfileMap) + V.push_back(std::make_pair(I.getKey(), &I.second)); + + std::stable_sort( + V.begin(), V.end(), + [](const NameFunctionSamples &A, const NameFunctionSamples &B) { + if (A.second->getTotalSamples() == B.second->getTotalSamples()) + return A.first > B.first; + return A.second->getTotalSamples() > B.second->getTotalSamples(); + }); + + for (const auto &I : V) { + if (std::error_code EC = write(*I.second)) + return EC; + } + return sampleprof_error::success; +} /// \brief Write samples to a text file. /// @@ -61,20 +95,21 @@ std::error_code SampleProfileWriterText::write(const FunctionSamples &S) { OS << "\n"; } - SampleSorter<LineLocation, FunctionSamples> SortedCallsiteSamples( + SampleSorter<LineLocation, FunctionSamplesMap> SortedCallsiteSamples( S.getCallsiteSamples()); Indent += 1; - for (const auto &I : SortedCallsiteSamples.get()) { - LineLocation Loc = I->first; - const FunctionSamples &CalleeSamples = I->second; - OS.indent(Indent); - if (Loc.Discriminator == 0) - OS << Loc.LineOffset << ": "; - else - OS << Loc.LineOffset << "." << Loc.Discriminator << ": "; - if (std::error_code EC = write(CalleeSamples)) - return EC; - } + for (const auto &I : SortedCallsiteSamples.get()) + for (const auto &FS : I->second) { + LineLocation Loc = I->first; + const FunctionSamples &CalleeSamples = FS.second; + OS.indent(Indent); + if (Loc.Discriminator == 0) + OS << Loc.LineOffset << ": "; + else + OS << Loc.LineOffset << "." << Loc.Discriminator << ": "; + if (std::error_code EC = write(CalleeSamples)) + return EC; + } Indent -= 1; return sampleprof_error::success; @@ -89,8 +124,7 @@ std::error_code SampleProfileWriterBinary::writeNameIdx(StringRef FName) { } void SampleProfileWriterBinary::addName(StringRef FName) { - auto NextIdx = NameTable.size(); - NameTable.insert(std::make_pair(FName, NextIdx)); + NameTable.insert(std::make_pair(FName, 0)); } void SampleProfileWriterBinary::addNames(const FunctionSamples &S) { @@ -102,11 +136,12 @@ void SampleProfileWriterBinary::addNames(const FunctionSamples &S) { } // Recursively add all the names for inlined callsites. - for (const auto &J : S.getCallsiteSamples()) { - const FunctionSamples &CalleeSamples = J.second; - addName(CalleeSamples.getName()); - addNames(CalleeSamples); - } + for (const auto &J : S.getCallsiteSamples()) + for (const auto &FS : J.second) { + const FunctionSamples &CalleeSamples = FS.second; + addName(CalleeSamples.getName()); + addNames(CalleeSamples); + } } std::error_code SampleProfileWriterBinary::writeHeader( @@ -127,10 +162,18 @@ std::error_code SampleProfileWriterBinary::writeHeader( addNames(I.second); } + // Sort the names to make NameTable is deterministic. + std::set<StringRef> V; + for (const auto &I : NameTable) + V.insert(I.first); + int i = 0; + for (const StringRef &N : V) + NameTable[N] = i++; + // Write out the name table. encodeULEB128(NameTable.size(), OS); - for (auto N : NameTable) { - OS << N.first; + for (auto N : V) { + OS << N; encodeULEB128(0, OS); } return sampleprof_error::success; @@ -180,14 +223,15 @@ std::error_code SampleProfileWriterBinary::writeBody(const FunctionSamples &S) { // Recursively emit all the callsite samples. encodeULEB128(S.getCallsiteSamples().size(), OS); - for (const auto &J : S.getCallsiteSamples()) { - LineLocation Loc = J.first; - const FunctionSamples &CalleeSamples = J.second; - encodeULEB128(Loc.LineOffset, OS); - encodeULEB128(Loc.Discriminator, OS); - if (std::error_code EC = writeBody(CalleeSamples)) - return EC; - } + for (const auto &J : S.getCallsiteSamples()) + for (const auto &FS : J.second) { + LineLocation Loc = J.first; + const FunctionSamples &CalleeSamples = FS.second; + encodeULEB128(Loc.LineOffset, OS); + encodeULEB128(Loc.Discriminator, OS); + if (std::error_code EC = writeBody(CalleeSamples)) + return EC; + } return sampleprof_error::success; } |