diff options
author | Scott Rifenbark <scott.m.rifenbark@intel.com> | 2011-12-07 13:57:53 -0800 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2011-12-16 16:58:08 +0000 |
commit | d122e37be3f339d628e5d82560811661a3e6c42b (patch) | |
tree | d39ce83450a5d57ba2fcfc2d7066f68d213adf0b /documentation | |
parent | 4ae4f1d553a48ee55c2406d519dfc2dee74d6d63 (diff) | |
download | ast2050-yocto-poky-d122e37be3f339d628e5d82560811661a3e6c42b.zip ast2050-yocto-poky-d122e37be3f339d628e5d82560811661a3e6c42b.tar.gz |
documentation/poky-ref-manual/usingpoky.xml: partial for YOCTO #1500
First draft of a re-write to the "Running a Build" section to try
and satisfy YOCTO #1500. I segmented the section into three areas
rather than a single area. This allowed me to create a sub-section
for the sstate stuff where it could be addressed on its own. I sent
the draft out to Richard and Mark H. and got feedback from RP that
is going to cause further changes. Thus, I am committing this partial
change.
(From yocto-docs rev: 4a99bcbda85ede0bd9490002c08461cc834b7518)
Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation')
-rw-r--r-- | documentation/poky-ref-manual/usingpoky.xml | 280 |
1 files changed, 233 insertions, 47 deletions
diff --git a/documentation/poky-ref-manual/usingpoky.xml b/documentation/poky-ref-manual/usingpoky.xml index d01cbaf..712dcc6 100644 --- a/documentation/poky-ref-manual/usingpoky.xml +++ b/documentation/poky-ref-manual/usingpoky.xml @@ -152,67 +152,253 @@ <title>Running a Build</title> <para> - You can find information on how to build an image using the Yocto Project in the + You can find general information on how to build an image using the + Yocto Project in the <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html#building-image'> Building an Image</ulink> section of the <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html'> Yocto Project Quick Start</ulink>. - This section provides a quick overview. + This section provides a summary of the build process and provides information + for less obvious aspects of the build process. </para> - <para> - The first thing you need to do is set up the Yocto Project build environment by sourcing - the environment setup script as follows: - <literallayout class='monospaced'> + <section id='build-overview'> + <title>Build Overview</title> + + <para> + The first thing you need to do is set up the Yocto Project build environment by sourcing + the environment setup script as follows: + <literallayout class='monospaced'> $ source oe-init-build-env [build_dir] - </literallayout> - </para> + </literallayout> + </para> - <para> - The <filename>build_dir</filename> is optional and specifies the directory Yocto Project - uses for the build. - If you do not specify a build directory it defaults to <filename>build</filename> - in your current working directory. - A common practice is to use a different build directory for different targets. - For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename> - target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target. - See <link linkend="structure-core-script">oe-init-build-env</link> - for more information on this script. - </para> + <para> + The <filename>build_dir</filename> is optional and specifies the directory Yocto Project + uses for the build. + If you do not specify a build directory it defaults to <filename>build</filename> + in your current working directory. + A common practice is to use a different build directory for different targets. + For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename> + target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target. + See <link linkend="structure-core-script">oe-init-build-env</link> + for more information on this script. + </para> - <para> - Once the Yocto Project build environment is set up, you can build a target using: - <literallayout class='monospaced'> + <para> + Once the Yocto Project build environment is set up, you can build a target using: + <literallayout class='monospaced'> $ bitbake <target> - </literallayout> - </para> + </literallayout> + </para> - <para> - The <filename>target</filename> is the name of the recipe you want to build. - Common targets are the images in <filename>meta/recipes-core/images</filename>, - <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project - files. - Or, the target can be the name of a recipe for a specific piece of software such as - <application>busybox</application>. - For more details about the images Yocto Project supports, see the - <link linkend="ref-images">'Reference: Images'</link> appendix. - </para> + <para> + The <filename>target</filename> is the name of the recipe you want to build. + Common targets are the images in <filename>meta/recipes-core/images</filename>, + <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project + files. + Or, the target can be the name of a recipe for a specific piece of software such as + <application>busybox</application>. + For more details about the images Yocto Project supports, see the + <link linkend="ref-images">'Reference: Images'</link> appendix. + </para> + + <note> + Building an image without GNU Public License Version 3 (GPLv3) components is + only supported for minimal and base images. + See <link linkend='ref-images'>'Reference: Images'</link> for more information. + </note> + </section> + + <section id='building-an-image-using-gpl-components'> + <title>Building an Image Using GPL Components</title> + + <para> + When building an image using GPL components, you need to maintain your original + settings and not switch back and forth applying different versions of the GNU + Public License. + If you rebuild using different versions of GPL, dependency errors might occur + due to some components not being rebuilt. + </para> + </section> - <note> - Building an image without GNU Public License Version 3 (GPLv3) components is - only supported for minimal and base images. - See <link linkend='ref-images'>'Reference: Images'</link> for more information. - </note> - - <note> - When building an image using GPL components, you need to maintain your original - settings and not switch back and forth applying different versions of the GNU - Public License. - If you rebuild using different versions of GPL, dependency errors might occur - due to some components not being rebuilt. - </note> + <section id="considering-shared-state-cache"> + <title>Considering Shared State Cache</title> + + <para> + By design, the Yocto Project builds everything from scratch unless it can determine that + a given task's inputs have not changed. + While building from scratch ensures that everything is current, it does also + mean that a lot of time could be spent rebuiding things that don't necessarily need built. + </para> + + <para> + The Yocto Project build process uses a shared state caching scheme to avoid having to + rebuild software when it is not necessary. + Because the build time for a Yocto image can be significant, it is helpful to try and + determine what really needs built and what can be skipped given a particular project's + build process. + </para> + + <para> + The scheme that the Yocto Project uses involves checksum generation and comparison for + a task's inputs. + The scheme also employs an area of memory called the shared state cache that is + pointed to by the <filename>SSTATE_DIR</filename> variable. + This area contains task output generated from a previous build. + If a given task's checksum matches the checksum of a previous build for the same + task, the build process uses the state of the cache rather than rerunning that + task. + </para> + + <para> + The previous paragraph is a simplistic explanation of how the build process + uses checksums and shared state memory cache to avoide building tasks that + don't need built. + If you want a bit more explanation on the topic, + see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared + State - What does it mean and why should I care?</ulink>" from the Yocto + Project discussion archives. + </para> + + <para> + As with all schemes, this one has some drawbacks. + It is possible that you could make implicit changes that are not factored into the checksum + calculation, but do affect a task's output. + A good example is perhaps when a tool changes its output. + Let's say that the output of <filename>rpmdeps</filename> needed to change. + The result of the change should be that all the "package", "package_write_rpm", + and "package_deploy-rpm" sstate-cache items would become invalid. + But, because this is a change that is external to the code and therefore implicit, + the associated sstate-cache items do not become invalidated. + In this case, the build process would use the cache items rather than running the + task again. + Obviously, these types of implicit changes can cause problems. + </para> + + <para> + To avoid these problems during the build, you need to understand the effects of any + change you make. + Note that any changes you make directly to a function automatically are factored into + the checksum calculation and thus, will invalidate the associated area of sstate cache. + You need to be aware of any implicit changes that are not obvious changes to the + code and could affect the output of a given task. + Once you are aware of such a change, you can take steps to invalidate the cache + and force the task to run. + The step to take is as simple as changing a function's comments in the source code. + For example, to invalidate package sstate files, change the comment statments + of <filename>do_package</filename> or one of the functions it calls. + The change is purely cosmetic, but it causes the checksum to be recalculated and + forces the task to be run again. + </para> + + <note> + For an example of a commit that makes a cosmetic change to invalidate an sstate, + see this + <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. + </note> + </section> + + +<!-- + + <section id="considering-shared-state-cache"> + <title>Considering Shared State Cache</title> + + <para> + What is shared state in general. + Benefits? + How we handle things + (reference https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm), + which is RP's dissertation on how YP solved it. + We need to talk a bit about checksum generation for tasks and how the + sstate code uses them to figure out what needs rebuilt and what can be re-loaded + from the sstate cache. + Need to tell about cases where an implicit change can mess things up and under + normal situations the state in the sstate cache would be used but it shouldn't be. + This is the scenario described by bug 1500 - typical case. + Then we talk about how we can invalidate parts of the cache on a per-class basis + if needed. + + there is a discussion at + https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm + that talks about sstate and how the YP team attacked and solved the problem. + This is probably a good place to get information from to broach the whole + sstate concept. + + YP, by default, builds from scratch. + This is good but it means spending a lot of time rebuilding things that don't + necessarily need rebuilding. + + The SSTATE_DIR variable points to the directory for the shared state cache that + is used during a build. + + A task's inputs have a checksum or signature associated with them. + If the checksum changes on an input as compared to a prior build, the task must be rerun. + The shared state (sstate) code keeps track of what output is generated by which tasks. + So if a task's inputs have not changed then the output associated with the task can + be yanked from some place and re-used. No re-build required for that particular task. + + A "run" shell script is created for each task. + You can create a checksum for the task based on the inputs to the task. + When you have this checksum, the code will look at it and compare it to the previous + checksum to see if the task's inputs have changed. + If so, the task needs to be re-run. + + Python tasks have python functions that access variables. + Python functions will call other python functions as well. + The solution was to figure out the variable and function dependencies and create + a checksum value for the data coming into the python task. + + Here is a conversation with Mark Hatle regarding bug 1500 (638 is related): + +(01:23:34 PM) scottrif: mark - you have a minute? +(01:34:05 PM) Mark Hatle: sure.. +(01:34:11 PM) Mark Hatle: might be a bit slow to respond, but I'm here +(01:34:45 PM) scottrif: Hi - I am looking at bug 1500 and trying to get a bit of better understanding. Here is the link to the bug - http://bugzilla.pokylinux.org/show_bug.cgi?id=1500 +(01:35:25 PM) scottrif: It seems that the key for the user here is to when to "Know" when to put some comments into a function to invalidate certain areas of sstate. +(01:35:49 PM) Mark Hatle: what the issue is, if you make changes to something that is not normally calculated in the checksums for sstate, then you can get package mismatches.. +(01:35:50 PM) scottrif: This trick of "knowing"... does it need to be explained? +(01:36:25 PM) Mark Hatle: The developer will have to know they made such a change.. Then to deal with this, they can use a patch like what is referenced to cause specific steps to be invalidated and various steps re-run.. +(01:37:01 PM) scottrif: so my question is will the developer know when they make a change like this? +(01:37:04 PM) Mark Hatle: In this case, we change part of the back-end packaging mechanisms.. which changed internal dependency generation. The sstate code does not checksum the internal dependency generation, it assumes that is code that doesn't change behavior +(01:37:24 PM) Mark Hatle: They should understand the ramifications of their changes — and thus know they need to do this. +(01:37:46 PM) Mark Hatle: Examples of times you need to do this. Back end packaging changes occur — i.e. you change the format of dependency generation.. +(01:38:38 PM) scottrif: do you have any other examples? +(01:38:39 PM) Mark Hatle: when you change a recipe itself, source code.. it is -not- necessary to do this +(01:38:49 PM) Mark Hatle: RP might be better at examples of when to do it.. +(01:39:11 PM) scottrif: right - If I change a recipe then every thing dependent further down the line gets regenerated right? +(01:39:17 PM) Mark Hatle: This should never be necessary when a recipe changes.. it will only be necessary when some classes or back-end (packaging frameworks) change.. +(01:39:21 PM) Mark Hatle: ya +(01:39:33 PM) Mark Hatle: Another way to think of this is implicit dependencies.. +(01:40:01 PM) Mark Hatle: I change RPM.. If you build something that produces an RPM package.. the assumption is the RPM package won't change, even if the RPM binary changes.. +(01:40:10 PM) Mark Hatle: If the format of the package changes.. you would need to do this +(01:40:53 PM) Mark Hatle: RP can probably give you an idea of the various implicit dependencies, and which ones this type of change is needed for +(01:41:26 PM) scottrif: okay. I am struggling a bit with how to word it. what I will do is write something up and send it out to you and RP for a look +(01:41:47 PM) Mark Hatle: ya, I understand.. it's an odd set of situations that can cause this — but we definitely need to document it +(01:42:01 PM) scottrif: I just want the information to help the user understand the conditions when they will want to invalidate parts of the sstate +(01:42:18 PM) scottrif: I will likely use the RPM example as the case to illustrate it +(01:42:26 PM) scottrif: as it seems pretty straight forward +(01:42:28 PM) Mark Hatle: yup. Key thing is it's only needed on implicit dependencies.. Normal case is back end packaging format changes.. +(01:42:31 PM) Mark Hatle: yup +(01:42:47 PM) scottrif: ok - thanks Mark + + Here is what RP wants to address 1500: + +If its desired to change the checksum of a given subset of tasks, maybe +due to a change which isn't directly visible in the code itself (e.g. a +tool changed its output) its possible to do this by changing a function +comments since the sstate checksums include the body of functions. To +invalidate package sstate files for example, do_package or one of the +functions it calls can be changed, even if its just a cosmetic change to +the commends. +http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54 +is an example of a commit which does this. + +--> </section> + <section id='usingpoky-install'> <title>Installing and Using the Result</title> |