From ed0a240e1632682ec4c33341f3e24ad71773cdfc Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Tue, 11 Dec 2012 12:07:58 -0600 Subject: documentation: Rename of poky-ref-manual folder to ref-manual. Changing the folder that holds the YP Reference Manual to be "ref-manual". This will help with confustion over the manual's intended purpose. (From yocto-docs rev: 1106442964b5080cb0b6b3bd3af32e9407c0f7c1) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- documentation/ref-manual/ref-bitbake.xml | 419 +++++++++++++++++++++++++++++++ 1 file changed, 419 insertions(+) create mode 100644 documentation/ref-manual/ref-bitbake.xml (limited to 'documentation/ref-manual/ref-bitbake.xml') diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml new file mode 100644 index 0000000..4d4b9d6 --- /dev/null +++ b/documentation/ref-manual/ref-bitbake.xml @@ -0,0 +1,419 @@ + %poky; ] > + + + + BitBake + + + BitBake is a program written in Python that interprets the metadata used by the OpenEmbedded + build system. + At some point, developers wonder what actually happens when you enter: + + $ bitbake core-image-sato + + + + + This chapter provides an overview of what happens behind the scenes from BitBake's perspective. + + + + BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships. + As such, it has no real knowledge of what the tasks being executed actually do. + BitBake just considers a list of tasks with dependencies and handles metadata + that consists of variables in a certain format that get passed to the tasks. + + +
+ Parsing + + + BitBake parses configuration files, classes, and .bb files. + + + + The first thing BitBake does is look for the bitbake.conf file. + This file resides in the + Source Directory + within the meta/conf/ directory. + BitBake finds it by examining its + BBPATH environment + variable and looking for the meta/conf/ + directory. + + + + The bitbake.conf file lists other configuration + files to include from a conf/ + directory below the directories listed in BBPATH. + In general, the most important configuration file from a user's perspective + is local.conf, which contains a user's customized + settings for the OpenEmbedded build environment. + Other notable configuration files are the distribution + configuration file (set by the + DISTRO variable) + and the machine configuration file + (set by the + MACHINE variable). + The DISTRO and MACHINE BitBake environment + variables are both usually set in + the local.conf file. + Valid distribution + configuration files are available in the meta/conf/distro/ directory + and valid machine configuration + files in the meta/conf/machine/ directory. + Within the meta/conf/machine/include/ + directory are various tune-*.inc configuration files that provide common + "tuning" settings specific to and shared between particular architectures and machines. + + + + After the parsing of the configuration files, some standard classes are included. + The base.bbclass file is always included. + Other classes that are specified in the configuration using the + INHERIT + variable are also included. + Class files are searched for in a classes subdirectory + under the paths in BBPATH in the same way as + configuration files. + + + + After classes are included, the variable + BBFILES + is set, usually in + local.conf, and defines the list of places to search for + .bb files. + By default, the BBFILES variable specifies the + meta/recipes-*/ directory within Poky. + Adding extra content to BBFILES is best achieved through the use of + BitBake layers as described in the + "Understanding and + Creating Layers" section of the Yocto Project Development Manual. + + + + BitBake parses each .bb file in BBFILES and + stores the values of various variables. + In summary, for each .bb + file the configuration plus the base class of variables are set, followed + by the data in the .bb file + itself, followed by any inherit commands that + .bb file might contain. + + + + Because parsing .bb files is a time + consuming process, a cache is kept to speed up subsequent parsing. + This cache is invalid if the timestamp of the .bb + file itself changes, or if the timestamps of any of the include, + configuration or class files the .bb + file depends on changes. + +
+ +
+ Preferences and Providers + + + Once all the .bb files have been + parsed, BitBake starts to build the target (core-image-sato + in the previous section's example) and looks for providers of that target. + Once a provider is selected, BitBake resolves all the dependencies for + the target. + In the case of core-image-sato, it would lead to + packagegroup-core-x11-sato, + which in turn leads to recipes like matchbox-terminal, + pcmanfm and gthumb. + These recipes in turn depend on eglibc and the toolchain. + + + + Sometimes a target might have multiple providers. + A common example is "virtual/kernel", which is provided by each kernel package. + Each machine often selects the best kernel provider by using a line similar to the + following in the machine configuration file: + + + + PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" + + + + The default PREFERRED_PROVIDER + is the provider with the same name as the target. + + + + Understanding how providers are chosen is made complicated by the fact + that multiple versions might exist. + BitBake defaults to the highest version of a provider. + Version comparisons are made using the same method as Debian. + You can use the + PREFERRED_VERSION + variable to specify a particular version (usually in the distro configuration). + You can influence the order by using the + DEFAULT_PREFERENCE + variable. + By default, files have a preference of "0". + Setting the DEFAULT_PREFERENCE to "-1" makes the + package unlikely to be used unless it is explicitly referenced. + Setting the DEFAULT_PREFERENCE to "1" makes it likely the package is used. + PREFERRED_VERSION overrides any DEFAULT_PREFERENCE setting. + DEFAULT_PREFERENCE is often used to mark newer and more experimental package + versions until they have undergone sufficient testing to be considered stable. + + + + In summary, BitBake has created a list of providers, which is prioritized, for each target. + +
+ +
+ Dependencies + + + Each target BitBake builds consists of multiple tasks such as + fetch, unpack, + patch, configure, + and compile. + For best performance on multi-core systems, BitBake considers each task as an independent + entity with its own set of dependencies. + + + + Dependencies are defined through several variables. + You can find information about variables BitBake uses in the BitBake documentation, + which is found in the bitbake/doc/manual directory within the + Source Directory. + At a basic level, it is sufficient to know that BitBake uses the + DEPENDS and + RDEPENDS variables when + calculating dependencies. + +
+ +
+ The Task List + + + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly what tasks it needs to run and in what + order it needs to run them. + The build now starts with BitBake forking off threads up to the limit set in the + BB_NUMBER_THREADS variable. + BitBake continues to fork threads as long as there are tasks ready to run, + those tasks have all their dependencies met, and the thread threshold has not been + exceeded. + + + + It is worth noting that you can greatly speed up the build time by properly setting + the BB_NUMBER_THREADS variable. + See the + "Building an Image" + section in the Yocto Project Quick Start for more information. + + + + As each task completes, a timestamp is written to the directory specified by the + STAMP variable. + On subsequent runs, BitBake looks within the /build/tmp/stamps + directory and does not rerun + tasks that are already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per + .bb file basis. + So, for example, if the configure stamp has a timestamp greater than the + compile timestamp for a given target, then the compile task would rerun. + Running the compile task again, however, has no effect on other providers + that depend on that target. + This behavior could change or become configurable in future versions of BitBake. + + + + Some tasks are marked as "nostamp" tasks. + No timestamp file is created when these tasks are run. + Consequently, "nostamp" tasks are always rerun. + +
+ +
+ Running a Task + + + Tasks can either be a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + ${WORKDIR}/temp/run.do_taskname.pid and then executes the script. + The generated shell script contains all the exported variables, and the shell functions + with all variables expanded. + Output from the shell script goes to the file ${WORKDIR}/temp/log.do_taskname.pid. + Looking at the expanded shell functions in the run file and the output in the log files + is a useful debugging technique. + + + + For Python tasks, BitBake executes the task internally and logs information to the + controlling terminal. + Future versions of BitBake will write the functions to files similar to the way + shell tasks are handled. + Logging will be handled in way similar to shell tasks as well. + + + + Once all the tasks have been completed BitBake exits. + + + + When running a task, BitBake tightly controls the execution environment + of the build tasks to make sure unwanted contamination from the build machine + cannot influence the build. + Consequently, if you do want something to get passed into the build + task's environment, you must take a few steps: + + Tell BitBake to load what you want from the environment + into the data store. + You can do so through the BB_ENV_EXTRAWHITE + variable. + For example, assume you want to prevent the build system from + accessing your $HOME/.ccache directory. + The following command tells BitBake to load + CCACHE_DIR from the environment into the data + store: + + export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR" + + Tell BitBake to export what you have loaded into the + environment store to the task environment of every running task. + Loading something from the environment into the data store + (previous step) only makes it available in the datastore. + To export it to the task environment of every running task, + use a command similar to the following in your + local.conf or distro configuration file: + + export CCACHE_DIR + + + + + + A side effect of the previous steps is that BitBake records the variable + as a dependency of the build process in things like the shared state + checksums. + If doing so results in unnecessary rebuilds of tasks, you can whitelist the + variable so that the shared state code ignores the dependency when it creates + checksums. + For information on this process, see the BB_HASHBASE_WHITELIST + example in the "Checksums (Signatures)" section. + +
+ +
+ BitBake Command Line + + + Following is the BitBake help output: + + + +$ bitbake --help +Usage: bitbake [options] [package ...] + +Executes the specified task (default is 'build') for a given set of BitBake files. +It expects that BBFILES is defined, which is a space separated list of files to +be executed. BBFILES does support wildcards. +Default BBFILES are the .bb files in the current directory. + +Options: + --version show program's version number and exit + -h, --help show this help message and exit + -b BUILDFILE, --buildfile=BUILDFILE + execute the task against this .bb file, rather than a + package from BBFILES. Does not handle any + dependencies. + -k, --continue continue as much as possible after an error. While the + target that failed, and those that depend on it, + cannot be remade, the other dependencies of these + targets can be processed all the same. + -a, --tryaltconfigs continue with builds by trying to use alternative + providers where possible. + -f, --force force run of specified cmd, regardless of stamp status + -c CMD, --cmd=CMD Specify task to execute. Note that this only executes + the specified task for the providee and the packages + it depends on, i.e. 'compile' does not implicitly call + stage for the dependencies (IOW: use only if you know + what you are doing). Depending on the base.bbclass a + listtasks tasks is defined and will show available + tasks + -r PREFILE, --read=PREFILE + read the specified file before bitbake.conf + -R POSTFILE, --postread=POSTFILE + read the specified file after bitbake.conf + -v, --verbose output more chit-chat to the terminal + -D, --debug Increase the debug level. You can specify this more + than once. + -n, --dry-run don't execute, just go through the motions + -S, --dump-signatures + don't execute, just dump out the signature + construction information + -p, --parse-only quit after parsing the BB files (developers only) + -s, --show-versions show current and preferred versions of all packages + -e, --environment show the global or per-package environment (this is + what used to be bbread) + -g, --graphviz emit the dependency trees of the specified packages in + the dot syntax + -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are already + provided (equivalent to ASSUME_PROVIDED). Useful to + make dependency graphs more appealing + -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS + Show debug logging for the specified logging domains + -P, --profile profile the command and print a report + -u UI, --ui=UI userinterface to use + -t SERVERTYPE, --servertype=SERVERTYPE + Choose which server to use, none, process or xmlrpc + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not + +
+ +
+ Fetchers + + + BitBake also contains a set of "fetcher" modules that allow + retrieval of source code from various types of sources. + For example, BitBake can get source code from a disk with the metadata, from websites, + from remote shell accounts or from Source Code Management (SCM) systems + like cvs/subversion/git. + + + + Fetchers are usually triggered by entries in + SRC_URI. + You can find information about the options and formats of entries for specific + fetchers in the BitBake manual located in the + bitbake/doc/manual directory of the + Source Directory. + + + + One useful feature for certain Source Code Manager (SCM) fetchers is the ability to + "auto-update" when the upstream SCM changes version. + Since this ability requires certain functionality from the SCM, not all + systems support it. + Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update". + This feature works using the SRCREV + variable. + See the + "Using an External SCM" section + in the Yocto Project Development Manual for more information. + + +
+ + + -- cgit v1.1