diff options
author | Scott Rifenbark <scott.m.rifenbark@intel.com> | 2014-01-29 12:06:15 -0600 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2014-02-11 12:17:06 +0000 |
commit | 3cdf800334bf7f6297e6c15bfc334814e8416380 (patch) | |
tree | 6a64a133528164fe04e2be45049c821e74cf6d8f /bitbake/doc/user-manual | |
parent | 89058e1ef7142765e29f5c403470d5bd2932e298 (diff) | |
download | ast2050-yocto-poky-3cdf800334bf7f6297e6c15bfc334814e8416380.zip ast2050-yocto-poky-3cdf800334bf7f6297e6c15bfc334814e8416380.tar.gz |
bitbake: user-manual-metadata.xml: Added "Checksums (Signatures)" section.
Added this section to the end of the Metadata chapter.
(Bitbake rev: b653c58284cafd0b79991520543ca6239705d36b)
Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'bitbake/doc/user-manual')
-rw-r--r-- | bitbake/doc/user-manual/user-manual-metadata.xml | 183 |
1 files changed, 183 insertions, 0 deletions
diff --git a/bitbake/doc/user-manual/user-manual-metadata.xml b/bitbake/doc/user-manual/user-manual-metadata.xml index d191d35..abfc89d 100644 --- a/bitbake/doc/user-manual/user-manual-metadata.xml +++ b/bitbake/doc/user-manual/user-manual-metadata.xml @@ -1092,4 +1092,187 @@ deletes all the flags for a variable. </para> </section> + + <section id='task-checksums-and-setscene'> + <title>Task Checksums and Setscene</title> + + <para> + This list is a place holder of content that needs explanation here. + Items should be moved to appropriate sections below as completed. + <itemizedlist> + <listitem><para><filename>STAMP</filename></para></listitem> + <listitem><para><filename>STAMPCLEAN</filename></para></listitem> + <listitem><para><filename>BB_STAMP_WHITELIST</filename></para></listitem> + <listitem><para><filename>BB_STAMP_POLICY</filename></para></listitem> + <listitem><para><filename>BB_HASHCHECK_FUNCTION</filename></para></listitem> + <listitem><para><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></para></listitem> + <listitem><para><filename>BB_SETSCENE_DEPVALID</filename></para></listitem> + <listitem><para><filename>BB_TASKHASH</filename></para></listitem> + </itemizedlist> + </para> + + <section id='checksums'> + <title>Checksums (Signatures)</title> + + <para> + BitBake uses checksums (or signatures) along with the setscene + to determine if a task needs to be run. + This section describes the process. + To help understand how BitBake does this, the section assumes an + OpenEmbedded metadata-based example. + </para> + + <para> + The setscene code uses a checksum, which is a unique signature of a task's + inputs, to determine if a task needs to be run again. + Because it is a change in a task's inputs that triggers a rerun, the process + needs to detect all the inputs to a given task. + For shell tasks, this turns out to be fairly easy because + BitBake generates a "run" shell script for each task and + it is possible to create a checksum that gives you a good idea of when + the task's data changes. + </para> + + <para> + To complicate the problem, some things should not be included in + the checksum. + First, there is the actual specific build path of a given task - + the working directory. + It does not matter if the work directory changes because it should not + affect the output for target packages. + The simplistic approach for excluding the work directory is to set + it to some fixed value and create the checksum for the "run" script. + </para> + + <para> + Another problem results from the "run" scripts containing functions that + might or might not get called. + The incremental build solution contains code that figures out dependencies + between shell functions. + This code is used to prune the "run" scripts down to the minimum set, + thereby alleviating this problem and making the "run" scripts much more + readable as a bonus. + </para> + + <para> + So far we have solutions for shell scripts. + What about Python tasks? + The same approach applies even though these tasks are more difficult. + The process needs to figure out what variables a Python function accesses + and what functions it calls. + Again, the incremental build solution contains code that first figures out + the variable and function dependencies, and then creates a checksum for the data + used as the input to the task. + </para> + + <para> + Like the working directory case, situations exist where dependencies + should be ignored. + For these cases, you can instruct the build process to ignore a dependency + by using a line like the following: + <literallayout class='monospaced'> + PACKAGE_ARCHS[vardepsexclude] = "MACHINE" + </literallayout> + This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not + depend on the value of <filename>MACHINE</filename>, even if it does reference it. + </para> + + <para> + Equally, there are cases where we need to add dependencies BitBake + is not able to find. + You can accomplish this by using a line like the following: + <literallayout class='monospaced'> + PACKAGE_ARCHS[vardeps] = "MACHINE" + </literallayout> + This example explicitly adds the <filename>MACHINE</filename> variable as a + dependency for <filename>PACKAGE_ARCHS</filename>. + </para> + + <para> + Consider a case with in-line Python, for example, where BitBake is not + able to figure out dependencies. + When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake + produces output when it discovers something for which it cannot figure out + dependencies. + </para> + + <para> + Thus far, this section has limited discussion to the direct inputs into a task. + Information based on direct inputs is referred to as the "basehash" in the + code. + However, there is still the question of a task's indirect inputs - the + things that were already built and present in the build directory. + The checksum (or signature) for a particular task needs to add the hashes + of all the tasks on which the particular task depends. + Choosing which dependencies to add is a policy decision. + However, the effect is to generate a master checksum that combines the basehash + and the hashes of the task's dependencies. + </para> + + <para> + At the code level, there are a variety of ways both the basehash and the + dependent task hashes can be influenced. + Within the BitBake configuration file, we can give BitBake some extra information + to help it construct the basehash. + The following statement effectively results in a list of global variable + dependency excludes - variables never included in any checksum. + This example uses variables from OpenEmbedded to help illustrate + the concept: + <literallayout class='monospaced'> + BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" + </literallayout> + The previous example excludes the work directory, which is part of + <filename>TMPDIR</filename>. + </para> + + <para> + The rules for deciding which hashes of dependent tasks to include through + dependency chains are more complex and are generally accomplished with a + Python function. + The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples + of this and also illustrates how you can insert your own policy into the system + if so desired. + This file defines the two basic signature generators OpenEmbedded Core + uses: "OEBasic" and "OEBasicHash". + By default, there is a dummy "noop" signature handler enabled in BitBake. + This means that behavior is unchanged from previous versions. + <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default + through this setting in the <filename>bitbake.conf</filename> file: + <literallayout class='monospaced'> + BB_SIGNATURE_HANDLER ?= "OEBasicHash" + </literallayout> + The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the + "OEBasic" version but adds the task hash to the stamp files. + This results in any metadata change that changes the task hash, automatically + causing the task to be run again. + This removes the need to bump + <link linkend='var-PR'><filename>PR</filename></link> + values, and changes to metadata automatically ripple across the build. + </para> + + <para> + It is also worth noting that the end result of these signature generators is to + make some dependency and hash information available to the build. + This information includes: + <itemizedlist> + <listitem><para><filename>BB_BASEHASH_task-<taskname></filename>: + The base hashes for each task in the recipe. + </para></listitem> + <listitem><para><filename>BB_BASEHASH_<filename:taskname></filename>: + The base hashes for each dependent task. + </para></listitem> + <listitem><para><filename>BBHASHDEPS_<filename:taskname></filename>: + The task dependencies for each task. + </para></listitem> + <listitem><para><filename>BB_TASKHASH</filename>: + The hash of the currently running task. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> </chapter> |