diff options
Diffstat (limited to 'documentation/ref-manual/ref-bitbake.xml')
-rw-r--r-- | documentation/ref-manual/ref-bitbake.xml | 419 |
1 files changed, 419 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml new file mode 100644 index 0000000..4d4b9d6 --- /dev/null +++ b/documentation/ref-manual/ref-bitbake.xml @@ -0,0 +1,419 @@ +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" +"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" +[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > + +<chapter id='ref-bitbake'> + + <title>BitBake</title> + + <para> + BitBake is a program written in Python that interprets the metadata used by the OpenEmbedded + build system. + At some point, developers wonder what actually happens when you enter: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para> + + <para> + This chapter provides an overview of what happens behind the scenes from BitBake's perspective. + </para> + + <note> + BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships. + As such, it has no real knowledge of what the tasks being executed actually do. + BitBake just considers a list of tasks with dependencies and handles metadata + that consists of variables in a certain format that get passed to the tasks. + </note> + + <section id='ref-bitbake-parsing'> + <title>Parsing</title> + + <para> + BitBake parses configuration files, classes, and <filename>.bb</filename> files. + </para> + + <para> + The first thing BitBake does is look for the <filename>bitbake.conf</filename> file. + This file resides in the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> + within the <filename>meta/conf/</filename> directory. + BitBake finds it by examining its + <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment + variable and looking for the <filename>meta/conf/</filename> + directory. + </para> + + <para> + The <filename>bitbake.conf</filename> file lists other configuration + files to include from a <filename>conf/</filename> + directory below the directories listed in <filename>BBPATH</filename>. + In general, the most important configuration file from a user's perspective + is <filename>local.conf</filename>, which contains a user's customized + settings for the OpenEmbedded build environment. + Other notable configuration files are the distribution + configuration file (set by the + <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable) + and the machine configuration file + (set by the + <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable). + The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment + variables are both usually set in + the <filename>local.conf</filename> file. + Valid distribution + configuration files are available in the <filename>meta/conf/distro/</filename> directory + and valid machine configuration + files in the <filename>meta/conf/machine/</filename> directory. + Within the <filename>meta/conf/machine/include/</filename> + directory are various <filename>tune-*.inc</filename> configuration files that provide common + "tuning" settings specific to and shared between particular architectures and machines. + </para> + + <para> + After the parsing of the configuration files, some standard classes are included. + The <filename>base.bbclass</filename> file is always included. + Other classes that are specified in the configuration using the + <filename><link linkend='var-INHERIT'>INHERIT</link></filename> + variable are also included. + Class files are searched for in a <filename>classes</filename> subdirectory + under the paths in <filename>BBPATH</filename> in the same way as + configuration files. + </para> + + <para> + After classes are included, the variable + <filename><link linkend='var-BBFILES'>BBFILES</link></filename> + is set, usually in + <filename>local.conf</filename>, and defines the list of places to search for + <filename>.bb</filename> files. + By default, the <filename>BBFILES</filename> variable specifies the + <filename>meta/recipes-*/</filename> directory within Poky. + Adding extra content to <filename>BBFILES</filename> is best achieved through the use of + BitBake layers as described in the + "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and + Creating Layers</ulink>" section of the Yocto Project Development Manual. + </para> + + <para> + BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and + stores the values of various variables. + In summary, for each <filename>.bb</filename> + file the configuration plus the base class of variables are set, followed + by the data in the <filename>.bb</filename> file + itself, followed by any inherit commands that + <filename>.bb</filename> file might contain. + </para> + + <para> + Because parsing <filename>.bb</filename> files is a time + consuming process, a cache is kept to speed up subsequent parsing. + This cache is invalid if the timestamp of the <filename>.bb</filename> + file itself changes, or if the timestamps of any of the include, + configuration or class files the <filename>.bb</filename> + file depends on changes. + </para> + </section> + + <section id='ref-bitbake-providers'> + <title>Preferences and Providers</title> + + <para> + Once all the <filename>.bb</filename> files have been + parsed, BitBake starts to build the target (<filename>core-image-sato</filename> + in the previous section's example) and looks for providers of that target. + Once a provider is selected, BitBake resolves all the dependencies for + the target. + In the case of <filename>core-image-sato</filename>, it would lead to + <filename>packagegroup-core-x11-sato</filename>, + which in turn leads to recipes like <filename>matchbox-terminal</filename>, + <filename>pcmanfm</filename> and <filename>gthumb</filename>. + These recipes in turn depend on <filename>eglibc</filename> and the toolchain. + </para> + + <para> + Sometimes a target might have multiple providers. + A common example is "virtual/kernel", which is provided by each kernel package. + Each machine often selects the best kernel provider by using a line similar to the + following in the machine configuration file: + </para> + + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" + </literallayout> + + <para> + The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename> + is the provider with the same name as the target. + </para> + + <para> + Understanding how providers are chosen is made complicated by the fact + that multiple versions might exist. + BitBake defaults to the highest version of a provider. + Version comparisons are made using the same method as Debian. + You can use the + <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename> + variable to specify a particular version (usually in the distro configuration). + You can influence the order by using the + <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename> + variable. + By default, files have a preference of "0". + Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the + package unlikely to be used unless it is explicitly referenced. + Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used. + <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting. + <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package + versions until they have undergone sufficient testing to be considered stable. + </para> + + <para> + In summary, BitBake has created a list of providers, which is prioritized, for each target. + </para> + </section> + + <section id='ref-bitbake-dependencies'> + <title>Dependencies</title> + + <para> + Each target BitBake builds consists of multiple tasks such as + <filename>fetch</filename>, <filename>unpack</filename>, + <filename>patch</filename>, <filename>configure</filename>, + and <filename>compile</filename>. + For best performance on multi-core systems, BitBake considers each task as an independent + entity with its own set of dependencies. + </para> + + <para> + Dependencies are defined through several variables. + You can find information about variables BitBake uses in the BitBake documentation, + which is found in the <filename>bitbake/doc/manual</filename> directory within the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. + At a basic level, it is sufficient to know that BitBake uses the + <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and + <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when + calculating dependencies. + </para> + </section> + + <section id='ref-bitbake-tasklist'> + <title>The Task List</title> + + <para> + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly what tasks it needs to run and in what + order it needs to run them. + The build now starts with BitBake forking off threads up to the limit set in the + <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable. + BitBake continues to fork threads as long as there are tasks ready to run, + those tasks have all their dependencies met, and the thread threshold has not been + exceeded. + </para> + + <para> + It is worth noting that you can greatly speed up the build time by properly setting + the <filename>BB_NUMBER_THREADS</filename> variable. + See the + "<ulink url='&YOCTO_DOCS_QS_URL;#building-image'>Building an Image</ulink>" + section in the Yocto Project Quick Start for more information. + </para> + + <para> + As each task completes, a timestamp is written to the directory specified by the + <filename><link linkend='var-STAMP'>STAMP</link></filename> variable. + On subsequent runs, BitBake looks within the <filename>/build/tmp/stamps</filename> + directory and does not rerun + tasks that are already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per + <filename>.bb</filename> file basis. + So, for example, if the configure stamp has a timestamp greater than the + compile timestamp for a given target, then the compile task would rerun. + Running the compile task again, however, has no effect on other providers + that depend on that target. + This behavior could change or become configurable in future versions of BitBake. + </para> + + <note> + Some tasks are marked as "nostamp" tasks. + No timestamp file is created when these tasks are run. + Consequently, "nostamp" tasks are always rerun. + </note> + </section> + + <section id='ref-bitbake-runtask'> + <title>Running a Task</title> + + <para> + Tasks can either be a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script. + The generated shell script contains all the exported variables, and the shell functions + with all variables expanded. + Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>. + Looking at the expanded shell functions in the run file and the output in the log files + is a useful debugging technique. + </para> + + <para> + For Python tasks, BitBake executes the task internally and logs information to the + controlling terminal. + Future versions of BitBake will write the functions to files similar to the way + shell tasks are handled. + Logging will be handled in way similar to shell tasks as well. + </para> + + <para> + Once all the tasks have been completed BitBake exits. + </para> + + <para> + When running a task, BitBake tightly controls the execution environment + of the build tasks to make sure unwanted contamination from the build machine + cannot influence the build. + Consequently, if you do want something to get passed into the build + task's environment, you must take a few steps: + <orderedlist> + <listitem><para>Tell BitBake to load what you want from the environment + into the data store. + You can do so through the <filename>BB_ENV_EXTRAWHITE</filename> + variable. + For example, assume you want to prevent the build system from + accessing your <filename>$HOME/.ccache</filename> directory. + The following command tells BitBake to load + <filename>CCACHE_DIR</filename> from the environment into the data + store: + <literallayout class='monospaced'> + export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR" + </literallayout></para></listitem> + <listitem><para>Tell BitBake to export what you have loaded into the + environment store to the task environment of every running task. + Loading something from the environment into the data store + (previous step) only makes it available in the datastore. + To export it to the task environment of every running task, + use a command similar to the following in your + <filename>local.conf</filename> or distro configuration file: + <literallayout class='monospaced'> + export CCACHE_DIR + </literallayout></para></listitem> + </orderedlist> + </para> + + <note> + A side effect of the previous steps is that BitBake records the variable + as a dependency of the build process in things like the shared state + checksums. + If doing so results in unnecessary rebuilds of tasks, you can whitelist the + variable so that the shared state code ignores the dependency when it creates + checksums. + For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename> + example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section. + </note> + </section> + + <section id='ref-bitbake-commandline'> + <title>BitBake Command Line</title> + + <para> + Following is the BitBake help output: + </para> + + <screen> +$ bitbake --help +Usage: bitbake [options] [package ...] + +Executes the specified task (default is 'build') for a given set of BitBake files. +It expects that BBFILES is defined, which is a space separated list of files to +be executed. BBFILES does support wildcards. +Default BBFILES are the .bb files in the current directory. + +Options: + --version show program's version number and exit + -h, --help show this help message and exit + -b BUILDFILE, --buildfile=BUILDFILE + execute the task against this .bb file, rather than a + package from BBFILES. Does not handle any + dependencies. + -k, --continue continue as much as possible after an error. While the + target that failed, and those that depend on it, + cannot be remade, the other dependencies of these + targets can be processed all the same. + -a, --tryaltconfigs continue with builds by trying to use alternative + providers where possible. + -f, --force force run of specified cmd, regardless of stamp status + -c CMD, --cmd=CMD Specify task to execute. Note that this only executes + the specified task for the providee and the packages + it depends on, i.e. 'compile' does not implicitly call + stage for the dependencies (IOW: use only if you know + what you are doing). Depending on the base.bbclass a + listtasks tasks is defined and will show available + tasks + -r PREFILE, --read=PREFILE + read the specified file before bitbake.conf + -R POSTFILE, --postread=POSTFILE + read the specified file after bitbake.conf + -v, --verbose output more chit-chat to the terminal + -D, --debug Increase the debug level. You can specify this more + than once. + -n, --dry-run don't execute, just go through the motions + -S, --dump-signatures + don't execute, just dump out the signature + construction information + -p, --parse-only quit after parsing the BB files (developers only) + -s, --show-versions show current and preferred versions of all packages + -e, --environment show the global or per-package environment (this is + what used to be bbread) + -g, --graphviz emit the dependency trees of the specified packages in + the dot syntax + -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are already + provided (equivalent to ASSUME_PROVIDED). Useful to + make dependency graphs more appealing + -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS + Show debug logging for the specified logging domains + -P, --profile profile the command and print a report + -u UI, --ui=UI userinterface to use + -t SERVERTYPE, --servertype=SERVERTYPE + Choose which server to use, none, process or xmlrpc + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not + </screen> + </section> + + <section id='ref-bitbake-fetchers'> + <title>Fetchers</title> + + <para> + BitBake also contains a set of "fetcher" modules that allow + retrieval of source code from various types of sources. + For example, BitBake can get source code from a disk with the metadata, from websites, + from remote shell accounts or from Source Code Management (SCM) systems + like <filename>cvs/subversion/git</filename>. + </para> + + <para> + Fetchers are usually triggered by entries in + <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>. + You can find information about the options and formats of entries for specific + fetchers in the BitBake manual located in the + <filename>bitbake/doc/manual</filename> directory of the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. + </para> + + <para> + One useful feature for certain Source Code Manager (SCM) fetchers is the ability to + "auto-update" when the upstream SCM changes version. + Since this ability requires certain functionality from the SCM, not all + systems support it. + Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update". + This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename> + variable. + See the + "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section + in the Yocto Project Development Manual for more information. + </para> + + </section> + +</chapter> +<!-- +vim: expandtab tw=80 ts=4 spell spelllang=en_gb +--> |