documentation: Rename of poky-ref-manual folder to ref-manual.

Changing the folder that holds the YP Reference Manual to be
"ref-manual".  This will help with confustion over the manual's
intended purpose.

(From yocto-docs rev: 1106442964b5080cb0b6b3bd3af32e9407c0f7c1)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 419 insertions, 0 deletions

diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
new file mode 100644
index 0000000..4d4b9d6
--- /dev/null
+++ b/documentation/ref-manual/ref-bitbake.xml
@@ -0,0 +1,419 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
+[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
+
+<chapter id='ref-bitbake'>
+
+    <title>BitBake</title>
+
+    <para>
+        BitBake is a program written in Python that interprets the metadata used by the OpenEmbedded
+        build system.
+        At some point, developers wonder what actually happens when you enter:
+        <literallayout class='monospaced'>
+     $ bitbake core-image-sato
+        </literallayout>
+    </para>
+
+    <para>
+        This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
+    </para>
+
+    <note>
+        BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
+        As such, it has no real knowledge of what the tasks being executed actually do.
+        BitBake just considers a list of tasks with dependencies and handles metadata
+        that consists of variables in a certain format that get passed to the tasks.
+    </note>
+
+    <section id='ref-bitbake-parsing'>
+        <title>Parsing</title>
+
+        <para>
+            BitBake parses configuration files, classes, and <filename>.bb</filename> files.
+        </para>
+
+        <para>
+            The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
+            This file resides in the
+            <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
+            within the <filename>meta/conf/</filename> directory.
+            BitBake finds it by examining its
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
+            variable and looking for the <filename>meta/conf/</filename>
+            directory.
+        </para>
+
+        <para>
+            The <filename>bitbake.conf</filename> file lists other configuration
+            files to include from a <filename>conf/</filename>
+            directory below the directories listed in <filename>BBPATH</filename>.
+            In general, the most important configuration file from a user's perspective
+            is <filename>local.conf</filename>, which contains a user's customized
+            settings for the OpenEmbedded build environment.
+            Other notable configuration files are the distribution
+            configuration file (set by the
+            <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
+            and the machine configuration file
+            (set by the
+            <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
+            The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
+            variables are both usually set in
+            the <filename>local.conf</filename> file.
+            Valid distribution
+            configuration files are available in the <filename>meta/conf/distro/</filename> directory
+            and valid machine configuration
+            files in the <filename>meta/conf/machine/</filename> directory.
+            Within the <filename>meta/conf/machine/include/</filename>
+            directory are various <filename>tune-*.inc</filename> configuration files that provide common
+            "tuning" settings specific to and shared between particular architectures and machines.
+        </para>
+
+        <para>
+            After the parsing of the configuration files, some standard classes are included.
+            The <filename>base.bbclass</filename> file is always included.
+            Other classes that are specified in the configuration using the
+            <filename><link linkend='var-INHERIT'>INHERIT</link></filename>
+            variable are also included.
+            Class files are searched for in a <filename>classes</filename> subdirectory
+            under the paths in <filename>BBPATH</filename> in the same way as
+            configuration files.
+        </para>
+
+        <para>
+            After classes are included, the variable
+            <filename><link linkend='var-BBFILES'>BBFILES</link></filename>
+            is set, usually in
+            <filename>local.conf</filename>, and defines the list of places to search for
+            <filename>.bb</filename> files.
+            By default, the <filename>BBFILES</filename> variable specifies the
+            <filename>meta/recipes-*/</filename> directory within Poky.
+            Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
+            BitBake layers as described in the
+            "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
+            Creating Layers</ulink>" section of the Yocto Project Development Manual.
+        </para>
+
+        <para>
+            BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
+            stores the values of various variables.
+            In summary, for each <filename>.bb</filename>
+            file the configuration plus the base class of variables are set, followed
+            by the data in the <filename>.bb</filename> file
+            itself, followed by any inherit commands that
+            <filename>.bb</filename> file might contain.
+        </para>
+
+        <para>
+            Because parsing <filename>.bb</filename> files is a time
+            consuming process, a cache is kept to speed up subsequent parsing.
+            This cache is invalid if the timestamp of the <filename>.bb</filename>
+            file itself changes, or if the timestamps of any of the include,
+            configuration or class files the <filename>.bb</filename>
+            file depends on changes.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-providers'>
+        <title>Preferences and Providers</title>
+
+        <para>
+            Once all the <filename>.bb</filename> files have been
+            parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
+            in the previous section's example) and looks for providers of that target.
+            Once a provider is selected, BitBake resolves all the dependencies for
+            the target.
+            In the case of <filename>core-image-sato</filename>, it would lead to
+            <filename>packagegroup-core-x11-sato</filename>,
+            which in turn leads to recipes like <filename>matchbox-terminal</filename>,
+            <filename>pcmanfm</filename> and <filename>gthumb</filename>.
+            These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
+        </para>
+
+        <para>
+            Sometimes a target might have multiple providers.
+            A common example is "virtual/kernel", which is provided by each kernel package.
+            Each machine often selects the best kernel provider by using a line similar to the
+            following in the machine configuration file:
+        </para>
+
+        <literallayout class='monospaced'>
+     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
+        </literallayout>
+
+        <para>
+            The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
+            is the provider with the same name as the target.
+        </para>
+
+        <para>
+            Understanding how providers are chosen is made complicated by the fact
+            that multiple versions might exist.
+            BitBake defaults to the highest version of a provider.
+            Version comparisons are made using the same method as Debian.
+            You can use the
+            <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
+            variable to specify a particular version (usually in the distro configuration).
+            You can influence the order by using the
+            <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
+            variable.
+            By default, files have a preference of "0".
+            Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
+            package unlikely to be used unless it is explicitly referenced.
+            Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
+            <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
+            <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
+            versions until they have undergone sufficient testing to be considered stable.
+        </para>
+
+        <para>
+            In summary, BitBake has created a list of providers, which is prioritized, for each target.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-dependencies'>
+        <title>Dependencies</title>
+
+        <para>
+            Each target BitBake builds consists of multiple tasks such as
+            <filename>fetch</filename>, <filename>unpack</filename>,
+            <filename>patch</filename>, <filename>configure</filename>,
+            and <filename>compile</filename>.
+            For best performance on multi-core systems, BitBake considers each task as an independent
+            entity with its own set of dependencies.
+        </para>
+
+        <para>
+            Dependencies are defined through several variables.
+            You can find information about variables BitBake uses in the BitBake documentation,
+            which is found in the <filename>bitbake/doc/manual</filename> directory within the
+            <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
+            At a basic level, it is sufficient to know that BitBake uses the
+            <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
+            <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when
+            calculating dependencies.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-tasklist'>
+        <title>The Task List</title>
+
+        <para>
+            Based on the generated list of providers and the dependency information,
+            BitBake can now calculate exactly what tasks it needs to run and in what
+            order it needs to run them.
+            The build now starts with BitBake forking off threads up to the limit set in the
+            <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
+            BitBake continues to fork threads as long as there are tasks ready to run,
+            those tasks have all their dependencies met, and the thread threshold has not been
+            exceeded.
+        </para>
+
+        <para>
+            It is worth noting that you can greatly speed up the build time by properly setting
+            the <filename>BB_NUMBER_THREADS</filename> variable.
+            See the
+            "<ulink url='&YOCTO_DOCS_QS_URL;#building-image'>Building an Image</ulink>"
+            section in the Yocto Project Quick Start for more information.
+        </para>
+
+        <para>
+            As each task completes, a timestamp is written to the directory specified by the
+            <filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
+            On subsequent runs, BitBake looks within the <filename>/build/tmp/stamps</filename>
+            directory and does not rerun
+            tasks that are already completed unless a timestamp is found to be invalid.
+            Currently, invalid timestamps are only considered on a per
+            <filename>.bb</filename> file basis.
+            So, for example, if the configure stamp has a timestamp greater than the
+            compile timestamp for a given target, then the compile task would rerun.
+            Running the compile task again, however, has no effect on other providers
+            that depend on that target.
+            This behavior could change or become configurable in future versions of BitBake.
+        </para>
+
+        <note>
+            Some tasks are marked as "nostamp" tasks.
+            No timestamp file is created when these tasks are run.
+            Consequently, "nostamp" tasks are always rerun.
+        </note>
+    </section>
+
+    <section id='ref-bitbake-runtask'>
+        <title>Running a Task</title>
+
+        <para>
+            Tasks can either be a shell task or a Python task.
+            For shell tasks, BitBake writes a shell script to
+            <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
+            The generated shell script contains all the exported variables, and the shell functions
+            with all variables expanded.
+            Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
+            Looking at the expanded shell functions in the run file and the output in the log files
+            is a useful debugging technique.
+        </para>
+
+        <para>
+            For Python tasks, BitBake executes the task internally and logs information to the
+            controlling terminal.
+            Future versions of BitBake will write the functions to files similar to the way
+            shell tasks are handled.
+            Logging will be handled in way similar to shell tasks as well.
+        </para>
+
+        <para>
+            Once all the tasks have been completed BitBake exits.
+        </para>
+
+        <para>
+            When running a task, BitBake tightly controls the execution environment
+            of the build tasks to make sure unwanted contamination from the build machine
+            cannot influence the build.
+            Consequently, if you do want something to get passed into the build
+            task's environment, you must take a few steps:
+            <orderedlist>
+                <listitem><para>Tell BitBake to load what you want from the environment
+                    into the data store.
+                    You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
+                    variable.
+                    For example, assume you want to prevent the build system from
+                    accessing your <filename>$HOME/.ccache</filename> directory.
+                    The following command tells BitBake to load
+                    <filename>CCACHE_DIR</filename> from the environment into the data
+                    store:
+                    <literallayout class='monospaced'>
+     export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
+                    </literallayout></para></listitem>
+                <listitem><para>Tell BitBake to export what you have loaded into the
+                    environment store to the task environment of every running task.
+                    Loading something from the environment into the data store
+                    (previous step) only makes it available in the datastore.
+                    To export it to the task environment of every running task,
+                    use a command similar to the following in your
+                    <filename>local.conf</filename> or distro configuration file:
+                    <literallayout class='monospaced'>
+     export CCACHE_DIR
+                    </literallayout></para></listitem>
+            </orderedlist>
+        </para>
+
+        <note>
+            A side effect of the previous steps is that BitBake records the variable
+            as a dependency of the build process in things like the shared state
+            checksums.
+            If doing so results in unnecessary rebuilds of tasks, you can whitelist the
+            variable so that the shared state code ignores the dependency when it creates
+            checksums.
+            For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
+            example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
+        </note>
+    </section>
+
+    <section id='ref-bitbake-commandline'>
+        <title>BitBake Command Line</title>
+
+        <para>
+            Following is the BitBake help output:
+        </para>
+
+        <screen>
+$ bitbake --help
+Usage: bitbake [options] [package ...]
+
+Executes the specified task (default is 'build') for a given set of BitBake files.
+It expects that BBFILES is defined, which is a space separated list of files to
+be executed.  BBFILES does support wildcards.
+Default BBFILES are the .bb files in the current directory.
+
+Options:
+  --version             show program's version number and exit
+  -h, --help            show this help message and exit
+  -b BUILDFILE, --buildfile=BUILDFILE
+                        execute the task against this .bb file, rather than a
+                        package from BBFILES. Does not handle any
+                        dependencies.
+  -k, --continue        continue as much as possible after an error. While the
+                        target that failed, and those that depend on it,
+                        cannot be remade, the other dependencies of these
+                        targets can be processed all the same.
+  -a, --tryaltconfigs   continue with builds by trying to use alternative
+                        providers where possible.
+  -f, --force           force run of specified cmd, regardless of stamp status
+  -c CMD, --cmd=CMD     Specify task to execute. Note that this only executes
+                        the specified task for the providee and the packages
+                        it depends on, i.e. 'compile' does not implicitly call
+                        stage for the dependencies (IOW: use only if you know
+                        what you are doing). Depending on the base.bbclass a
+                        listtasks tasks is defined and will show available
+                        tasks
+  -r PREFILE, --read=PREFILE
+                        read the specified file before bitbake.conf
+  -R POSTFILE, --postread=POSTFILE
+                        read the specified file after bitbake.conf
+  -v, --verbose         output more chit-chat to the terminal
+  -D, --debug           Increase the debug level. You can specify this more
+                        than once.
+  -n, --dry-run         don't execute, just go through the motions
+  -S, --dump-signatures
+                        don't execute, just dump out the signature
+                        construction information
+  -p, --parse-only      quit after parsing the BB files (developers only)
+  -s, --show-versions   show current and preferred versions of all packages
+  -e, --environment     show the global or per-package environment (this is
+                        what used to be bbread)
+  -g, --graphviz        emit the dependency trees of the specified packages in
+                        the dot syntax
+  -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
+                        Assume these dependencies don't exist and are already
+                        provided (equivalent to ASSUME_PROVIDED). Useful to
+                        make dependency graphs more appealing
+  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
+                        Show debug logging for the specified logging domains
+  -P, --profile         profile the command and print a report
+  -u UI, --ui=UI        userinterface to use
+  -t SERVERTYPE, --servertype=SERVERTYPE
+                        Choose which server to use, none, process or xmlrpc
+  --revisions-changed   Set the exit code depending on whether upstream
+                        floating revisions have changed or not
+        </screen>
+    </section>
+
+    <section id='ref-bitbake-fetchers'>
+        <title>Fetchers</title>
+
+        <para>
+            BitBake also contains a set of "fetcher" modules that allow
+            retrieval of source code from various types of sources.
+            For example, BitBake can get source code from a disk with the metadata, from websites,
+            from remote shell accounts or from Source Code Management (SCM) systems
+            like <filename>cvs/subversion/git</filename>.
+        </para>
+
+        <para>
+            Fetchers are usually triggered by entries in
+            <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
+            You can find information about the options and formats of entries for specific
+            fetchers in the BitBake manual located in the
+            <filename>bitbake/doc/manual</filename> directory of the
+            <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
+        </para>
+
+        <para>
+            One useful feature for certain Source Code Manager (SCM) fetchers is the ability to
+            "auto-update" when the upstream SCM changes version.
+            Since this ability requires certain functionality from the SCM, not all
+            systems support it.
+            Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update".
+            This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
+            variable.
+            See the
+            "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section
+            in the Yocto Project Development Manual for more information.
+        </para>
+
+    </section>
+
+</chapter>
+<!--
+vim: expandtab tw=80 ts=4 spell spelllang=en_gb
+-->