bitbake: user-manual: Review edits from Richard (second draft)

Applied the comprehensive set of review comments from Richard
Purdie.  All files affected.  One major point here was that the
"BitBake Command" chapter was eliminated.  This information was
folded into various areas of the book.  Consequently, the bits
including the file for make had to be updated.

(Bitbake rev: 8ec38c6b456a92a0e0b9b04c2793a5b148be5027)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

7 files changed, 579 insertions, 417 deletions

diff --git a/bitbake/doc/user-manual/user-manual-bitbakecommand.xml b/bitbake/doc/user-manual/user-manual-bitbakecommand.xml
deleted file mode 100644
index 5c301a5..0000000
--- a/bitbake/doc/user-manual/user-manual-bitbakecommand.xml
+++ /dev/null
@@ -1,341 +0,0 @@
-<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
-"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
-
-<chapter id="user-manual-command">
-    <title>The BitBake Command</title>
-
-    <para>
-        BitBake is the underlying piece of the build system.
-        Two excellent examples are the Yocto Project and the OpenEmbedded
-        build systems.
-        Each provide an environment in which to develop embedded Linux
-        images, and each use BitBake as their underlying build engine.
-    </para>
-
-    <para>
-        BitBake facilitates executing tasks in a single <filename>.bb</filename>
-        file, or executing a given task on a set of multiple
-        <filename>.bb</filename> files, accounting for interdependencies
-        amongst them.
-        This chapter presents the BitBake syntax, provides some execution
-        examples, and shows you how to control BitBake with key metadata.
-    </para>
-
-    <section id='usage-and-syntax'>
-        <title>Usage and syntax</title>
-
-        <para>
-            Following is the usage and syntax for BitBake:
-            <literallayout class='monospaced'>
-     $ bitbake -h
-Usage: bitbake [options] [recipename/target ...]
-
-    Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
-    It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
-    will provide the layer, BBFILES and other configuration information.
-
-Options:
-  --version             show program's version number and exit
-  -h, --help            show this help message and exit
-  -b BUILDFILE, --buildfile=BUILDFILE
-                        Execute tasks from a specific .bb recipe directly.
-                        WARNING: Does not handle any dependencies from other
-                        recipes.
-  -k, --continue        Continue as much as possible after an error. While the
-                        target that failed and anything depending on it cannot
-                        be built, as much as possible will be built before
-                        stopping.
-  -a, --tryaltconfigs   Continue with builds by trying to use alternative
-                        providers where possible.
-  -f, --force           Force the specified targets/task to run (invalidating
-                        any existing stamp file).
-  -c CMD, --cmd=CMD     Specify the task to execute. The exact options
-                        available depend on the metadata. Some examples might
-                        be 'compile' or 'populate_sysroot' or 'listtasks' may
-                        give a list of the tasks available.
-  -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
-                        Invalidate the stamp for the specified task such as
-                        'compile' and then run the default task for the
-                        specified target(s).
-  -r PREFILE, --read=PREFILE
-                        Read the specified file before bitbake.conf.
-  -R POSTFILE, --postread=POSTFILE
-                        Read the specified file after bitbake.conf.
-  -v, --verbose         Output more log message data to the terminal.
-  -D, --debug           Increase the debug level. You can specify this more
-                        than once.
-  -n, --dry-run         Don't execute, just go through the motions.
-  -S, --dump-signatures
-                        Don't execute, just dump out the signature
-                        construction information.
-  -p, --parse-only      Quit after parsing the BB recipes.
-  -s, --show-versions   Show current and preferred versions of all recipes.
-  -e, --environment     Show the global or per-package environment complete
-                        with information about where variables were
-                        set/changed.
-  -g, --graphviz        Save dependency tree information for the specified
-                        targets in the dot syntax.
-  -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
-                        Assume these dependencies don't exist and are already
-                        provided (equivalent to ASSUME_PROVIDED). Useful to
-                        make dependency graphs more appealing
-  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
-                        Show debug logging for the specified logging domains
-  -P, --profile         Profile the command and save reports.
-  -u UI, --ui=UI        The user interface to use (e.g. knotty, hob, depexp).
-  -t SERVERTYPE, --servertype=SERVERTYPE
-                        Choose which server to use, process or xmlrpc.
-  --revisions-changed   Set the exit code depending on whether upstream
-                        floating revisions have changed or not.
-  --server-only         Run bitbake without a UI, only starting a server
-                        (cooker) process.
-  -B BIND, --bind=BIND  The name/address for the bitbake server to bind to.
-  --no-setscene         Do not run any setscene tasks. sstate will be ignored
-                        and everything needed, built.
-  --remote-server=REMOTE_SERVER
-                        Connect to the specified server.
-  -m, --kill-server     Terminate the remote server.
-  --observe-only        Connect to a server as an observing-only client.
-  --status-only         Check the status of the remote bitbake server.
-
-            </literallayout>
-        </para>
-    </section>
-
-    <section id='bitbake-examples'>
-        <title>Examples</title>
-
-        <para>
-            This section presents some examples showing how to use BitBake.
-        </para>
-
-        <section id='example-executing-a-task-against-a-single-recipe'>
-            <title>Executing a Task Against a Single Recipe</title>
-
-            <para>
-                Executing tasks for a single recipe file is relatively simple.
-                You specify the file in question, and BitBake parses
-                it and executes the specified task.
-                If you do not specify a task, BitBake executes the default
-                task, which is "build”.
-                BitBake obeys inter-task dependencies when doing
-                so.
-            </para>
-
-            <para>
-                The following command runs the clean task on the
-                <filename>foo_1.0.bb</filename> recipe file:
-                <literallayout class='monospaced'>
-     $ bitbake -b foo.bb -c clean
-                </literallayout>
-                The following command runs the build task, which is
-                the default task, on the <filename>foo_1.0.bb</filename>
-                recipe file:
-                <literallayout class='monospaced'>
-     $ bitbake -b foo_1.0.bb
-                </literallayout>
-            </para>
-        </section>
-
-        <section id='executing-tasks-against-a-set-of-recipe-files'>
-            <title>Executing Tasks Against a Set of Recipe Files</title>
-
-            <para>
-                There are a number of additional complexities introduced
-                when one wants to manage multiple <filename>.bb</filename>
-                files.
-                Clearly there needs to be a way to tell BitBake what
-                files are available, and of those, which you
-                want to execute.
-                There also needs to be a way for each recipe
-                to express its dependencies, both for build-time and
-                runtime.
-                There must be a way for you to express recipe preferences
-                when multiple recipes provide the same functionality, or when
-                there are multiple versions of a  recipe.
-            </para>
-
-            <para>
-                The <filename>bitbake</filename> command, when not using
-                "--buildfile" or "-b" only accepts a "PROVIDER".
-                You cannot provide anything else.
-                By default, a recipe file generally "PROVIDES" its
-                "packagename", "packagename-version", and
-                "packagename-version-revision" as shown in the following
-                example:
-                <literallayout class='monospaced'>
-     $ bitbake foo
-
-     $ bitbake foo-1.0
-
-     $ bitbake foo-1.0-r0
-                </literallayout>
-                This next example "PROVIDES" the package name and also uses
-                the "-c" option to tell BitBake to just excute the
-                <filename>do_clean</filename> task:
-                <literallayout class='monospaced'>
-     $ bitbake -c clean foo
-                </literallayout>
-            </para>
-        </section>
-
-        <section id='generating-dependency-graphs'>
-            <title>Generating Dependency Graphs</title>
-
-            <para>
-                BitBake is able to generate dependency graphs using
-                the dot syntax.
-                You can convert these graphs into images using the dot
-                application from
-                <ulink url='http://www.graphviz.org'>Graphviz</ulink>.
-            </para>
-
-            <para>
-                When you generate a dependency graph, BitBake writes two files
-                to the current working directory:
-                <filename>depends.dot</filename>, which contains dependency information
-                at the package level, and <filename>task-depends.dot</filename>,
-                which contains a breakdown of the dependencies at the task level.
-            </para>
-
-            <para>
-                To stop depending on common depends, use use the "-I" depend
-                option and BitBake omits them from the graph.
-                Leaving this information out can produce more readable graphs.
-                This way, you can remove from the graph
-                <filename>DEPENDS</filename> from inherited classes
-                such as <filename>base.bbclass</filename>.
-            </para>
-
-            <para>
-                Here are two exmples that create dependency graphs.
-                The second example omits common depends from the graph:
-                <literallayout class='monospaced'>
-     $ bitbake -g foo
-
-     $ bitbake -g -I virtual/whatever -I bloom foo
-                </literallayout>
-            </para>
-        </section>
-    </section>
-
-    <section id='controlling-bitbake'>
-        <title>Controlling BitBake</title>
-
-        <para>
-            Including variables in your recipe and class files help control
-            how BitBake operates.
-        </para>
-
-        <section id='execution-threads'>
-            <title>Execution Threads</title>
-
-            <para>
-                You can control how many thread BitBake supports by using the
-                <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
-                variable.
-                You would set this in your <filename>local.conf</filename>
-                configuration file.
-            </para>
-        </section>
-
-        <section id='using-provides'>
-            <title>Using <filename>PROVIDES</filename></title>
-
-            <para>
-                This example shows the usage of the
-                <filename>PROVIDES</filename> variable, which allows a
-                given <filename>.bb</filename> to specify what
-                functionality it provides.
-                <literallayout class='monospaced'>
-     package1.bb:
-
-     PROVIDES += "virtual/package"
-
-     package2.bb:
-
-     DEPENDS += "virtual/package"
-
-     package3.bb:
-
-     PROVIDES += "virtual/package"
-                </literallayout>
-                As you can see, we have two different
-                recipes that provide the same functionality
-                (virtual/package).
-                Clearly, there needs to be a way for the person running
-                BitBake to control which of those providers
-                gets used.
-                There is, indeed, such a way.
-            </para>
-
-            <para>
-                The following would go into a <filename>.conf</filename>
-                file, to select package1:
-                <literallayout class='monospaced'>
-     PREFERRED_PROVIDER_virtual/package = "package1"
-                </literallayout>
-            </para>
-        </section>
-
-        <section id='specifying-version-preference'>
-            <title>Specifying Version Preference</title>
-
-            <para>
-                When there are multiple “versions” of a given package,
-                BitBake defaults to selecting the most recent
-                version, unless otherwise specified.
-                If the <filename>.bb</filename> in question has a
-                <filename>DEFAULT_PREFERENCE</filename> set lower than
-                the other recipes (default is 0), then it will not be
-                selected.
-                This allows the person or persons maintaining
-                the repository of <filename>.bb</filename> files to specify
-                their preference for the default selected version.
-                In addition, the user can specify their preferred version.
-            </para>
-
-            <para>
-                If the first <filename>.bb</filename> is named
-                <filename>a_1.1.bb</filename>, then the
-                <filename>PN</filename> variable will be set to
-                “a”, and the <filename>PV</filename> variable will be
-                set to 1.1.
-            </para>
-
-            <para>
-                If we then have an <filename>a_1.2.bb</filename>, BitBake
-                will choose 1.2 by default.
-                However, if we define the following variable in a
-                <filename>.conf</filename> file that BitBake parses, we
-                can change that.
-                <literallayout class='monospaced'>
-     PREFERRED_VERSION_a = "1.1"
-                </literallayout>
-            </para>
-        </section>
-
-        <section id='using-recipe-file-collections'>
-            <title>Using Recipe File Collections</title>
-
-            <para>
-                Recipe file collections exist to allow the user to
-                have multiple repositories of
-                <filename>.bb</filename> files that contain the same
-                exact package.
-                For example, one could easily use them to make one's
-                own local copy of an upstream repository, but with
-                custom modifications that one does not want upstream.
-                Here is an example:
-                <literallayout class='monospaced'>
-    BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
-    BBFILE_COLLECTIONS = "upstream local"
-    BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
-    BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
-    BBFILE_PRIORITY_upstream = "5"
-    BBFILE_PRIORITY_local = "10"
-                </literallayout>
-            </para>
-        </section>
-    </section>
-</chapter>
diff --git a/bitbake/doc/user-manual/user-manual-execution.xml b/bitbake/doc/user-manual/user-manual-execution.xml
index e9f19be..148ac3e 100644
--- a/bitbake/doc/user-manual/user-manual-execution.xml
+++ b/bitbake/doc/user-manual/user-manual-execution.xml
@@ -23,11 +23,19 @@
      $ bitbake &lt;target&gt;
         </literallayout>
         For information on the BitBake command and its options,
-        see the
-        "<link linkend='user-manual-command'>BitBake Command</link>
-        chapter.
+        see
+        "<link linkend='user-manual-command'>The BitBake Command</link>"
+        section.
     </para>
 
+    <note>
+        Prior to executing BitBake, you should take advantage of parallel
+        thread execution by setting the
+        <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+        variable in your <filename>local.conf</filename>
+        configuration file.
+    </note>
+
     <section id='parsing-the-base-configuration-metadata'>
         <title>Parsing the Base Configuration Metadata</title>
 
@@ -103,10 +111,13 @@
                 <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
                 <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
                 <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
-                <listitem><para><link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link></para></listitem>
-                <listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
-                <listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
+                <listitem><para>
+                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
+                    </para></listitem>
             </itemizedlist>
+            You can find information on how to pass environment variables into the BitBake
+            execution environment in the
+            "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>" section.
         </para>
 
         <para>
@@ -146,12 +157,16 @@
         <para>
             Only variable definitions and include directives are allowed
             in <filename>.conf</filename> files.
-            The following variables include:
+            Some variables directly influence BitBake's behavior.
+            These variables might have been set from the environment
+            depending on the environment variables previously
+            mentioned or set in the configuration files.
+            See the
+            "<link linkend='ref-variables-glos'>Variables Glossary</link>"
+            for a full list of variables.
+            The following list shows common variables set:
             <itemizedlist>
                 <listitem><para>
-                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
-                    </para></listitem>
-                <listitem><para>
                     <link linkend='var-BBDEBUG'><filename>BBDEBUG</filename></link>
                     </para></listitem>
                 <listitem><para>
@@ -196,6 +211,8 @@
                 <listitem><para>
                     <link linkend='var-BBMASK'><filename>BBMASK</filename></link>
                     </para></listitem>
+                <listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
+                <listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
             </itemizedlist>
         </para>
 
@@ -234,8 +251,7 @@
         <title>Locating and Parsing Recipes</title>
 
         <para>
-            During the configuration phase, BitBake will have
-            set
+            During the configuration phase, BitBake will have set
             <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
             BitBake now uses it to construct a list of recipes to parse,
             along with any append files (<filename>.bbappend</filename>)
@@ -244,7 +260,7 @@
             available files and supports wildcards.
             An example would be:
             <literallayout class='monospaced'>
-     BBFILES = "/path/to/bbfiles/*.bb"
+     BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
             </literallayout>
             BitBake parses each recipe and append file located
             with <filename>BBFILES</filename> and stores the values of
@@ -279,7 +295,8 @@
         <para>
             By the time parsing is complete for a recipe, BitBake
             has a list of tasks that the recipe defines and a set of
-            data consisting of keys and values.
+            data consisting of keys and values as well as
+            dependency information about the tasks.
         </para>
 
         <para>
@@ -287,16 +304,48 @@
             It only needs a small subset of the information to make
             decisions about the recipe.
             Consequently, BitBake caches the values in which it is
-            interested.
-        </para>
-
-        <para>
-            Subsequent BitBake commands then parse the base
-            configuration and compute a checksum of that data.
-            If that checksum matches what is in the cache, the
-            recipe and class files have not changed.
-            In this case, BitBake reloads the cached information
-            about the recipe instead of reparsing it from scratch.
+            interested and does not store the rest of the information.
+            Experience has shown it's faster to re-parse the metadata than to
+            try and write it out to the disk and reload then it.
+        </para>
+
+        <para>
+            Where possible, subsequent BitBake commands reuse this cache of
+            recipe information.
+            The validity of this cache is determined by first computing a
+            checksum of the base configuration data (see
+            <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
+            and then checking if the checksum matches.
+            If that checksum matches what is in the cache and the recipe
+            and class files have not changed, Bitbake is able to use
+            the cache.
+            BitBake then reloads the cached information about the recipe
+            instead of reparsing it from scratch.
+        </para>
+
+        <para>
+            Recipe file collections exist to allow the user to
+            have multiple repositories of
+            <filename>.bb</filename> files that contain the same
+            exact package.
+            For example, one could easily use them to make one's
+            own local copy of an upstream repository, but with
+            custom modifications that one does not want upstream.
+            Here is an example:
+            <literallayout class='monospaced'>
+    BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
+    BBFILE_COLLECTIONS = "upstream local"
+    BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
+    BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
+    BBFILE_PRIORITY_upstream = "5"
+    BBFILE_PRIORITY_local = "10"
+            </literallayout>
+            <note>
+                The layers mechanism is now the preferred method of collecting
+                code.
+                While the collections code remains, its main use is to set layer
+                priorities and to deal with overlap (conflicts) between layers.
+            </note>
         </para>
     </section>
 
@@ -304,38 +353,60 @@
         <title>Preferences and Providers</title>
 
         <para>
-            Assuming BitBake has been instructed to execute a target and
-            that all the recipe files have been parsed, BitBake starts to
-            build the target and look for providers of that target.
-            Once a provider is selected, BitBake resolves all the dependencies for
-            the target.
-            As an example, suppose the target is
-            <filename>core-image-sato</filename>.
-            In this case, it would lead to
-            <filename>packagegroup-core-x11-sato</filename>,
-            which in turn leads to recipes like <filename>matchbox-terminal</filename>,
-            <filename>pcmanfm</filename> and <filename>gthumb</filename>.
-            These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
+            Assuming BitBake has been instructed to execute a target
+            and that all the recipe files have been parsed, BitBake
+            starts to figure out how to build the target.
+            BitBake starts by looking through the
+            <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
+            set in recipe files.
+            The default <filename>PROVIDES</filename> for a recipe is its name
+            (<link linkend='var-PN'><filename>PN</filename></link>),
+            however, a recipe can provide multiple things.
         </para>
 
         <para>
-            Sometimes a target might have multiple providers.
-            A common example is "virtual/kernel", which is provided by each kernel package.
-            Each machine often selects the best kernel provider by using a line similar to the
-            following in the machine configuration file:
+            As an example of adding an extra provider, suppose a recipe named
+            <filename>package1.bb</filename> contained the following:
+            <literallayout class='monospaced'>
+     PROVIDES += "virtual/package"
+            </literallayout>
+            The recipe now provides both "package1" and "virtual/package.
+            The "virtual/" namespace is often used to denote cases where
+            multiple providers are expected with the user choosing between
+            them.
+            Kernels and toolchain components are common cases of this in
+            OpenEmbedded.
         </para>
 
-        <literallayout class='monospaced'>
-     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
-        </literallayout>
-
         <para>
+            Sometimes a target might have multiple providers.
+            A common example is "virtual/kernel", which is provided by each
+            kernel recipe.
+            Each machine often selects the best kernel provider by using a
+            line similar to the following in the machine configuration file:
+            <literallayout class='monospaced'>
+     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
+            </literallayout>
             The default
             <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
             is the provider with the same name as the target.
         </para>
 
         <para>
+            Bitbake iterates through each target it needs to build and resolve
+            them using this process.
+            As an example, suppose the target is
+            <filename>core-image-sato</filename>.
+            In this case, it would lead to
+            <filename>packagegroup-core-x11-sato</filename>,
+            which in turn leads to recipes like
+            <filename>matchbox-terminal</filename>, <filename>pcmanfm</filename>
+            and <filename>gthumb</filename>.
+            These recipes in turn depend on <filename>eglibc</filename>
+            and the toolchain.
+        </para>
+
+        <para>
             Understanding how providers are chosen is made complicated by the fact
             that multiple versions might exist.
             BitBake defaults to the highest version of a provider.
@@ -358,6 +429,41 @@
         <para>
             In summary, BitBake has created a list of providers, which is prioritized, for each target.
         </para>
+
+        <para>
+            When there are multiple “versions” of a given package,
+            BitBake defaults to selecting the most recent
+            version, unless otherwise specified.
+            If the recipe in question has a
+            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+            set lower than
+            the other recipes (default is 0), then it will not be
+            selected.
+            This allows the person or persons maintaining
+            the repository of recipe files to specify
+            their preference for the default selected version.
+            In addition, the user can specify their preferred version.
+        </para>
+
+        <para>
+            If the first recipe is named <filename>a_1.1.bb</filename>,
+            then the
+            <link linkend='var-PN'><filename>PN</filename></link> variable
+            will be set to “a”, and the
+            <link linkend='var-PV'><filename>PV</filename></link>
+            variable will be set to 1.1.
+        </para>
+
+        <para>
+            If we then have a recipe named <filename>a_1.2.bb</filename>, BitBake
+            will choose 1.2 by default.
+            However, if we define the following variable in a
+            <filename>.conf</filename> file that BitBake parses, we
+            can change that.
+            <literallayout class='monospaced'>
+     PREFERRED_VERSION_a = "1.1"
+            </literallayout>
+        </para>
     </section>
 
     <section id='bb-bitbake-dependencies'>
@@ -422,7 +528,20 @@
             compile timestamp for a given target, then the compile task would rerun.
             Running the compile task again, however, has no effect on other providers
             that depend on that target.
-            This behavior could change or become configurable in future versions of BitBake.
+        </para>
+
+        <para>
+            The exact format of the stamps is partly configurable.
+            In modern versions of BitBake, a hash is appended to the
+            stamp so that if the configuration changes, the stamp becomes
+            invalid and the task is automatically rerun.
+            This hash, or signature used, is governed by the signature policy
+            that is configured (see the
+            "<link linkend='checksums'>Checksums (Signatures)</link>"
+            section for information).
+            It is also possible to append extra metadata to the stamp using
+            the "stamp-extra-info" task flag.
+            For example, OpenEmbedded uses this flag to make some tasks machine-specific.
         </para>
 
         <note>
@@ -462,7 +581,12 @@
         </para>
 
         <para>
-            Variables specific to scheduling functionality exist:
+            The order in which BitBake runs the tasks is controlled by its
+            task scheduler.
+            It is possible to configure the scheduler and define custom
+            implementations for specific use cases.
+            For more information, see these variables that control the
+            behavior:
             <itemizedlist>
                 <listitem><para>
                     <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
@@ -471,22 +595,10 @@
                     <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
                     </para></listitem>
             </itemizedlist>
-        </para>
-    </section>
-
-    <section id='setscene'>
-        <title>Setscene</title>
-
-        <para>
-            This section needs to get the concept of the setscene across.
-            The reader needs to know what it is and what it is used for during
-            the build process.
-        </para>
-
-        <para>
-            You can find more information on setscene metadata in the
-            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
-            section.
+            It is possible to have functions run before and after a task's main
+            function.
+            This is done using the "prefuncs" and "postfuncs" flags of the task
+            that lists the functions to run.
         </para>
     </section>
 
@@ -495,10 +607,10 @@
 
         <para>
             A checksum is a unique signature of a task's inputs.
-            The setscene code uses a checksum to determine if a task needs
-            to be run.
+            The signature of a task can be used to determine if a task
+            needs to be run.
             Because it is a change in a task's inputs that triggers running
-            the task, the process needs to detect all the inputs to a given task.
+            the task, BitBake needs to detect all the inputs to a given task.
             For shell tasks, this turns out to be fairly easy because
             BitBake generates a "run" shell script for each task and
             it is possible to create a checksum that gives you a good idea of when
@@ -514,6 +626,10 @@
             affect the output for target packages.
             The simplistic approach for excluding the working directory is to set
             it to some fixed value and create the checksum for the "run" script.
+            BitBake goes one step better and uses the
+            <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
+            variable to define a list of variables that should never be included
+            when generating the signatures.
         </para>
 
         <para>
@@ -652,4 +768,87 @@
             section.
         </para>
     </section>
+
+    <section id='setscene'>
+        <title>Setscene</title>
+
+        <para>
+            The setscene process enables BitBake to handle "pre-built" artifacts.
+            The ability to handle and reuse these artifacts allows BitBake
+            the luxury of not having to build something from scratch every time.
+            Instead, BitBake can use, when possible, existing build artifacts.
+        </para>
+
+        <para>
+            BitBake needs to have reliable data indicating whether or not an
+            artifact is compatible.
+            Signatures, described in the previous section, provide an ideal
+            way of representing whether an artifact is compatible.
+            If a signature is the same, an object can be reused.
+        </para>
+
+        <para>
+            If an object can be reused, the problem then becomes how to
+            replace a given task or set of tasks with the pre-built artifact.
+            BitBake solves the problem with the "setscene" process.
+        </para>
+
+        <para>
+            When BitBake is asked to build a given target, before building anything,
+            it first asks whether cached information is available for any of the
+            targets it's building, or any of the intermediate targets.
+            If cached information is available, BitBake uses this information instead of
+            running the main tasks.
+        </para>
+
+        <para>
+            BitBake first calls the function defined by the
+            <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
+            variable with a list of tasks and corresponding
+            hashes it wants to build.
+            This function is designed to be fast and returns a list
+            of the tasks for which it believes in can obtain artifacts.
+        </para>
+
+        <para>
+            Next, for each of the tasks that were returned as possibilities,
+            BitBake executes a setscene version of the task that the possible
+            artifact covers.
+            Setscene versions of a task have the string "_setscene" appended to the
+            task name.
+            So, for example, the task with the name <filename>xxx</filename> has
+            a setscene task named <filename>xxx_setscene</filename>.
+            The setscene version of the task executes and provides the necessary
+            artifacts returning either success or failure.
+            <note>
+                Artifacts might need to be fetched from the network.
+            </note>
+        </para>
+
+        <para>
+            As previously mentioned, an artifact can cover more than one task.
+            For example, it is pointless to obtain a compiler if you
+            already have the compiled binary.
+            To handle this, BitBake calls the
+            <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
+            function for each successful setscene task to know whether or not it needs
+            to obtain the dependencies of that task.
+        </para>
+
+        <para>
+            Finally, after all the setscene tasks have executed, BitBake calls the
+            function listed in
+            <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link>
+            with the list of tasks BitBake thinks has been "covered".
+            The metadata can then ensure that this list is correct and can
+            inform BitBake that it wants specific tasks to be run regardless
+            of the setscene result.
+        </para>
+
+        <para>
+            You can find more information on setscene metadata in the
+            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+            section.
+        </para>
+    </section>
 </chapter>
diff --git a/bitbake/doc/user-manual/user-manual-fetching.xml b/bitbake/doc/user-manual/user-manual-fetching.xml
index 87951fd..b4c1aa2 100644
--- a/bitbake/doc/user-manual/user-manual-fetching.xml
+++ b/bitbake/doc/user-manual/user-manual-fetching.xml
@@ -31,7 +31,7 @@
             perhaps in a specific way.
             Getting and unpacking the files is often optionally followed
             by patching.
-            Patching, however, is not covered by the fetch.
+            Patching, however, is not covered by this module.
         </para>
 
         <para>
diff --git a/bitbake/doc/user-manual/user-manual-intro.xml b/bitbake/doc/user-manual/user-manual-intro.xml
index c1a9aed..6f9ad20 100644
--- a/bitbake/doc/user-manual/user-manual-intro.xml
+++ b/bitbake/doc/user-manual/user-manual-intro.xml
@@ -322,6 +322,29 @@
                 Information in append files overrides the information in the
                 similarly-named recipe file.
             </para>
+
+            <para>
+                When you name an append file, you can use the
+                wildcard character (%) to allow for matching recipe names.
+                For example, suppose you have an append file named
+                as follows:
+                <literallayout class='monospaced'>
+     busybox_1.21.%.bbappend
+                </literallayout>
+                That append file would match any <filename>busybox_1.21.x.bb</filename>
+                version of the recipe.
+                So, the append file would match the following recipe names:
+                <literallayout class='monospaced'>
+     busybox_1.21.1.bb
+     busybox_1.21.2.bb
+     busybox_1.21.3.bb
+                </literallayout>
+                If the <filename>busybox</filename> recipe was updated to
+                <filename>busybox_1.3.0.bb</filename>, the append name would not
+                match.
+                However, if you named the append file
+                <filename>busybox_1.%.bb</filename>, then you would have a match.
+            </para>
         </section>
     </section>
 
@@ -373,7 +396,13 @@
                 <listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis>
                     Downloading a snapshot of BitBake from the
                     source code repository gives you access to a known
-                    branch or release of BitBake.</para>
+                    branch or release of BitBake.
+                    <note>
+                         Cloning the Git repository, as described earlier,
+                         is the preferred method for getting BitBake.
+                         Cloning the repository makes it easier to update as
+                         patches are added to the stable branches.
+                    </note></para>
                     <para>The following example downloads a snapshot of
                     BitBake version 1.17.0:
                     <literallayout class='monospaced'>
@@ -387,4 +416,223 @@
             </itemizedlist>
         </para>
     </section>
+
+    <section id="user-manual-command">
+        <title>The BitBake Command</title>
+
+        <para>
+            BitBake is the underlying piece of the build system.
+            Two excellent examples are the Yocto Project and the OpenEmbedded
+            build systems.
+            Each provide an environment in which to develop embedded Linux
+            images, and each use BitBake as their underlying build engine.
+        </para>
+
+        <para>
+            BitBake facilitates executing tasks in a single <filename>.bb</filename>
+            file, or executing a given task on a set of multiple
+            <filename>.bb</filename> files, accounting for interdependencies
+            amongst them.
+            This section presents the BitBake syntax and provides some execution
+            examples.
+        </para>
+
+        <section id='usage-and-syntax'>
+            <title>Usage and syntax</title>
+
+            <para>
+                Following is the usage and syntax for BitBake:
+                <literallayout class='monospaced'>
+     $ bitbake -h
+Usage: bitbake [options] [recipename/target ...]
+
+    Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
+    It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
+    will provide the layer, BBFILES and other configuration information.
+
+Options:
+  --version             show program's version number and exit
+  -h, --help            show this help message and exit
+  -b BUILDFILE, --buildfile=BUILDFILE
+                        Execute tasks from a specific .bb recipe directly.
+                        WARNING: Does not handle any dependencies from other
+                        recipes.
+  -k, --continue        Continue as much as possible after an error. While the
+                        target that failed and anything depending on it cannot
+                        be built, as much as possible will be built before
+                        stopping.
+  -a, --tryaltconfigs   Continue with builds by trying to use alternative
+                        providers where possible.
+  -f, --force           Force the specified targets/task to run (invalidating
+                        any existing stamp file).
+  -c CMD, --cmd=CMD     Specify the task to execute. The exact options
+                        available depend on the metadata. Some examples might
+                        be 'compile' or 'populate_sysroot' or 'listtasks' may
+                        give a list of the tasks available.
+  -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
+                        Invalidate the stamp for the specified task such as
+                        'compile' and then run the default task for the
+                        specified target(s).
+  -r PREFILE, --read=PREFILE
+                        Read the specified file before bitbake.conf.
+  -R POSTFILE, --postread=POSTFILE
+                        Read the specified file after bitbake.conf.
+  -v, --verbose         Output more log message data to the terminal.
+  -D, --debug           Increase the debug level. You can specify this more
+                        than once.
+  -n, --dry-run         Don't execute, just go through the motions.
+  -S, --dump-signatures
+                        Don't execute, just dump out the signature
+                        construction information.
+  -p, --parse-only      Quit after parsing the BB recipes.
+  -s, --show-versions   Show current and preferred versions of all recipes.
+  -e, --environment     Show the global or per-package environment complete
+                        with information about where variables were
+                        set/changed.
+  -g, --graphviz        Save dependency tree information for the specified
+                        targets in the dot syntax.
+  -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
+                        Assume these dependencies don't exist and are already
+                        provided (equivalent to ASSUME_PROVIDED). Useful to
+                        make dependency graphs more appealing
+  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
+                        Show debug logging for the specified logging domains
+  -P, --profile         Profile the command and save reports.
+  -u UI, --ui=UI        The user interface to use (e.g. knotty, hob, depexp).
+  -t SERVERTYPE, --servertype=SERVERTYPE
+                        Choose which server to use, process or xmlrpc.
+  --revisions-changed   Set the exit code depending on whether upstream
+                        floating revisions have changed or not.
+  --server-only         Run bitbake without a UI, only starting a server
+                        (cooker) process.
+  -B BIND, --bind=BIND  The name/address for the bitbake server to bind to.
+  --no-setscene         Do not run any setscene tasks. sstate will be ignored
+                        and everything needed, built.
+  --remote-server=REMOTE_SERVER
+                        Connect to the specified server.
+  -m, --kill-server     Terminate the remote server.
+  --observe-only        Connect to a server as an observing-only client.
+  --status-only         Check the status of the remote bitbake server.
+
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='bitbake-examples'>
+            <title>Examples</title>
+
+            <para>
+                This section presents some examples showing how to use BitBake.
+            </para>
+
+            <section id='example-executing-a-task-against-a-single-recipe'>
+                <title>Executing a Task Against a Single Recipe</title>
+
+                <para>
+                    Executing tasks for a single recipe file is relatively simple.
+                    You specify the file in question, and BitBake parses
+                    it and executes the specified task.
+                    If you do not specify a task, BitBake executes the default
+                    task, which is "build”.
+                    BitBake obeys inter-task dependencies when doing
+                    so.
+                </para>
+
+                <para>
+                    The following command runs the clean task on the
+                    <filename>foo_1.0.bb</filename> recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo.bb -c clean
+                    </literallayout>
+                    The following command runs the build task, which is
+                    the default task, on the <filename>foo_1.0.bb</filename>
+                    recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo_1.0.bb
+                    </literallayout>
+                </para>
+            </section>
+
+            <section id='executing-tasks-against-a-set-of-recipe-files'>
+                <title>Executing Tasks Against a Set of Recipe Files</title>
+
+                <para>
+                    There are a number of additional complexities introduced
+                    when one wants to manage multiple <filename>.bb</filename>
+                    files.
+                    Clearly there needs to be a way to tell BitBake what
+                    files are available, and of those, which you
+                    want to execute.
+                    There also needs to be a way for each recipe
+                    to express its dependencies, both for build-time and
+                    runtime.
+                    There must be a way for you to express recipe preferences
+                    when multiple recipes provide the same functionality, or when
+                    there are multiple versions of a  recipe.
+                </para>
+
+                <para>
+                    The <filename>bitbake</filename> command, when not using
+                    "--buildfile" or "-b" only accepts a "PROVIDER".
+                    You cannot provide anything else.
+                    By default, a recipe file generally "PROVIDES" its
+                    "packagename", "packagename-version", and
+                    "packagename-version-revision" as shown in the following
+                    example:
+                    <literallayout class='monospaced'>
+     $ bitbake foo
+
+     $ bitbake foo-1.0
+
+     $ bitbake foo-1.0-r0
+                    </literallayout>
+                    This next example "PROVIDES" the package name and also uses
+                    the "-c" option to tell BitBake to just execute the
+                    <filename>do_clean</filename> task:
+                    <literallayout class='monospaced'>
+     $ bitbake -c clean foo
+                    </literallayout>
+                </para>
+            </section>
+
+            <section id='generating-dependency-graphs'>
+                <title>Generating Dependency Graphs</title>
+
+                <para>
+                    BitBake is able to generate dependency graphs using
+                    the dot syntax.
+                    You can convert these graphs into images using the dot
+                    application from
+                    <ulink url='http://www.graphviz.org'>Graphviz</ulink>.
+                </para>
+
+                <para>
+                    When you generate a dependency graph, BitBake writes two files
+                    to the current working directory:
+                    <filename>depends.dot</filename>, which contains dependency information
+                    at the package level, and <filename>task-depends.dot</filename>,
+                    which contains a breakdown of the dependencies at the task level.
+                </para>
+
+                <para>
+                    To stop depending on common depends, use use the "-I" depend
+                    option and BitBake omits them from the graph.
+                    Leaving this information out can produce more readable graphs.
+                    This way, you can remove from the graph
+                    <filename>DEPENDS</filename> from inherited classes
+                    such as <filename>base.bbclass</filename>.
+                </para>
+
+                <para>
+                    Here are two examples that create dependency graphs.
+                    The second example omits common depends from the graph:
+                    <literallayout class='monospaced'>
+     $ bitbake -g foo
+
+     $ bitbake -g -I virtual/whatever -I bloom foo
+                    </literallayout>
+                </para>
+            </section>
+        </section>
+    </section>
 </chapter>
diff --git a/bitbake/doc/user-manual/user-manual-metadata.xml b/bitbake/doc/user-manual/user-manual-metadata.xml
index cabf25f..9cd8bf0 100644
--- a/bitbake/doc/user-manual/user-manual-metadata.xml
+++ b/bitbake/doc/user-manual/user-manual-metadata.xml
@@ -819,6 +819,20 @@
             </para>
         </section>
 
+        <section id='deleting-a-task'>
+            <title>Deleting a Task</title>
+
+            <para>
+                As well as being able to add tasks, tasks can also be deleted.
+                This is done simply with <filename>deltask</filename> command.
+                For example, to delete the example task used in the previous
+                sections, you would use:
+                <literallayout class='monospaced'>
+     deltask printdate
+                </literallayout>
+            </para>
+        </section>
+
         <section id='passing-information-into-the-build-task-environment'>
             <title>Passing Information Into the Build Task Environment</title>
 
@@ -867,6 +881,28 @@
                         </note></para></listitem>
                 </orderedlist>
             </para>
+
+            <para>
+                Sometimes, its useful to be able to obtain information
+                from the original execution environment.
+                Bitbake saves a copy of the original environment into
+                a special variable named
+                <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
+            </para>
+
+            <para>
+                The <filename>BB_ORIGENV</filename> variable returns a datastore
+                object that can be queried using the standard datastore operators
+                such as <filename>getVar()</filename>.
+                The datastore object is useful, for example, to find the original
+                <filename>DISPLAY</filename> variable.
+            </para>
+
+            <para>
+                By default, BitBake cleans the environment to include only those
+                things exported or listed in its whitelist to ensure that the build
+                environment is reproducible and consistent.
+            </para>
         </section>
     </section>
 
@@ -975,6 +1011,17 @@
                     "<link linkend='inter-task-dependencies'>Inter-Task Dependencies</link>"
                     section for more information.
                     </para></listitem>
+                <listitem><para><emphasis>postfuncs:</emphasis>
+                    List of functions to call after the completion of the task.
+                    </para></listitem>
+                <listitem><para><emphasis>prefuncs:</emphasis>
+                    List of functions to call before the task executes.
+                    </para></listitem>
+                <listitem><para><emphasis>stamp-extra-info:</emphasis>
+                    Extra stamp information to append to the task's stamp
+                    As an example, OpenEmbedded uses this flag to allow
+                    machine-specific tasks.
+                    </para></listitem>
             </itemizedlist>
         </para>
     </section>
@@ -1016,7 +1063,7 @@
         </para>
 
         <para>
-            During all builds, the following common events occur:
+            During a standard build, the following common events might occur:
             <itemizedlist>
                 <listitem><para>
                     <filename>bb.event.ConfigParsed()</filename>
@@ -1100,7 +1147,19 @@
             <link linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
             and
             <link linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
-            variables:
+            variables.
+            <note>
+                The mechanism for this class extension is extremely
+                specific to the implementation.
+                Usually, the recipe's
+                <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
+                <link linkend='var-PN'><filename>PN</filename></link>, and
+                <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+                variables would need to be modified by the extension class.
+                For specific examples, see the OE-Core
+                <filename>native</filename>, <filename>nativesdk</filename>,
+                and <filename>multilib</filename> classes.
+            </note>
             <itemizedlist>
                 <listitem><para><emphasis><filename>BBCLASSEXTEND</filename>:</emphasis>
                     This variable is a space separated list of classes used to "extend" the
diff --git a/bitbake/doc/user-manual/user-manual-ref-variables.xml b/bitbake/doc/user-manual/user-manual-ref-variables.xml
index e1bf2b5..ff2d59a 100644
--- a/bitbake/doc/user-manual/user-manual-ref-variables.xml
+++ b/bitbake/doc/user-manual/user-manual-ref-variables.xml
@@ -388,16 +388,15 @@
         <glossentry id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
             <glossdef>
                 <para>
-                    Lists variables that are excluded from checksum
-                    comparisons to determine if the cache can be reused.
+                    Lists variables that are excluded from base configuration
+                    checksum, which is used to determine if the cache can
+                    be reused.
                 </para>
 
                 <para>
                     One of the ways BitBake determines whether to re-parse the
                     main metadata is through checksums of the variables in the
-                    datastore of the base configuration data (see the
-                    <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
-                    variable).
+                    datastore of the base configuration data.
                     There are variables that you typically want to exclude when
                     checking whether or not to re-parse and thus rebuild the
                     cache.
diff --git a/bitbake/doc/user-manual/user-manual.xml b/bitbake/doc/user-manual/user-manual.xml
index 76c3edf..9f94886 100644
--- a/bitbake/doc/user-manual/user-manual.xml
+++ b/bitbake/doc/user-manual/user-manual.xml
@@ -81,8 +81,6 @@
 
     <xi:include href="user-manual-fetching.xml"/>
 
-    <xi:include href="user-manual-bitbakecommand.xml"/>
-
     <xi:include href="user-manual-ref-variables.xml"/>
 
     <xi:include href="user-manual-hello.xml"/>