documenation/poky-ref-manual/technical-details.xml: Shared State

I updated the tips and tricks section to include two sub-sections.
The first if the debugging stuff from Richard's email.  The
second section is how to invalidate a section of the sstate
cache on a per-class basis.

Fixes [YOCTO #1500]

(From yocto-docs rev: 7493f5e7ec641e7cd79f8e3e6c045cadf7a6999c)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 77 insertions, 102 deletions

diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
index 1657431..dfb5b72 100644
--- a/documentation/poky-ref-manual/technical-details.xml
+++ b/documentation/poky-ref-manual/technical-details.xml
@@ -493,111 +493,86 @@
 
         <para>
             The code in the Yocto Project that supports incremental builds is not 
-            simple code. 
-            Consequently, when things go wrong, debugging needs to be straightforward. 
-            Because of this, the Yocto Project team included strong debugging
-            tools.
-        </para>
-
-        <para>
-            First, whenever a shared state package is written out, so is a
-            corresponding <filename>.siginfo</filename> file. 
-            This practice results in a pickled python database of all
-            the metadata that went into creating the hash for a given shared state
-            package.
-        </para>
-
-        <para>
-            Second, if BitBake is run with the <filename>--dump-signatures</filename>
-            (or <filename>-S</filename>) option, BitBake dumps out 
-            <filename>.siginfo</filename> files in
-            the stamp directory for every task it would have executed instead of
-            building the target package specified.
-        </para>
-
-        <para>
-            Finally, there is a <filename>bitbake-diffsigs</filename> command that
-            can process these <filename>.siginfo</filename> files. 
-            If one file is specified, it will dump out the dependency
-            information in the file. 
-            If two files are specified, it will compare the
-            two files and dump out the differences between the two.
-            This allows the question of "What changed between X and Y?" to be
-            answered easily.
-        </para>
+            simple code.
+            This section presents some tips and tricks that help you work around 
+            issues related to shared state code.
+        </para>
+
+        <section id='debugging'>
+            <title>Debugging</title>
+
+            <para>
+                When things go wrong, debugging needs to be straightforward. 
+                Because of this, the Yocto Project team included strong debugging
+                tools:
+                <itemizedlist>
+                    <listitem><para>Whenever a shared state package is written out, so is a
+                        corresponding <filename>.siginfo</filename> file. 
+                        This practice results in a pickled python database of all
+                        the metadata that went into creating the hash for a given shared state
+                        package.</para></listitem>
+                    <listitem><para>If BitBake is run with the <filename>--dump-signatures</filename>
+                        (or <filename>-S</filename>) option, BitBake dumps out 
+                        <filename>.siginfo</filename> files in
+                        the stamp directory for every task it would have executed instead of
+                        building the target package specified.</para></listitem>
+                    <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
+                        can process these <filename>.siginfo</filename> files. 
+                        If one file is specified, it will dump out the dependency
+                        information in the file. 
+                        If two files are specified, it will compare the
+                        two files and dump out the differences between the two.
+                        This allows the question of "What changed between X and Y?" to be
+                        answered easily.</para></listitem>
+                </itemizedlist>
+            </para>
+        </section>
+
+        <section id='invalidating-shared-state'>
+            <title>Invalidating Shared State</title>
+
+            <para>
+                The Yocto Project shared state code uses checksums and shared state memory 
+                cache to avoide building tasks that don't need built.
+                As with all schemes, this one has some drawbacks.
+                It is possible that you could make implicit changes that are not factored into the checksum 
+                calculation, but do affect a task's output. 
+                A good example is perhaps when a tool changes its output.
+                Let's say that the output of <filename>rpmdeps</filename> needed to change.
+                The result of the change should be that all the "package", "package_write_rpm",
+                and "package_deploy-rpm" shared state cache items would become invalid.
+                But, because this is a change that is external to the code and therefore implicit,
+                the associated shared state cache items do not become invalidated.
+                In this case, the build process would use the cache items rather than running the 
+                task again. 
+                Obviously, these types of implicit changes can cause problems.
+            </para>
+
+            <para>
+                To avoid these problems during the build, you need to understand the effects of any
+                change you make.
+                Note that any changes you make directly to a function automatically are factored into 
+                the checksum calculation and thus, will invalidate the associated area of sstate cache.
+                You need to be aware of any implicit changes that are not obvious changes to the 
+                code and could affect the output of a given task. 
+                Once you are aware of such a change, you can take steps to invalidate the cache 
+                and force the task to run. 
+                The step to take is as simple as changing a function's comments in the source code. 
+                For example, to invalidate package shared state files, change the comment statments
+                of <filename>do_package</filename> or the comments of one of the functions it calls.
+                The change is purely cosmetic, but it causes the checksum to be recalculated and  
+                forces the task to be run again.
+            </para>
+
+            <note>
+                For an example of a commit that makes a cosmetic change to invalidate an sstate, 
+                see this
+                <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
+            </note>
+        </section>                
     </section>
 </section>
 
-<!--
-
-    <para>
-        The Yocto Project build process uses a shared state caching scheme to avoid having to 
-        rebuild software when it is not necessary.  
-        Because the build time for a Yocto image can be significant, it is helpful to try and 
-        determine what really needs built and what can be skipped given a particular project's
-        build process.
-    </para>
-
-    <para>
-        The scheme that the Yocto Project uses involves checksum generation and comparison for
-        a task's inputs.
-        The scheme also employs an area of memory called the shared state cache that is 
-        pointed to by the <filename>SSTATE_DIR</filename> variable.
-        This area contains task output generated from a previous build.  
-        If a given task's checksum matches the checksum of a previous build for the same  
-        task, the build process uses the state of the cache rather than rerunning that 
-        task.
-    </para>
-
-    <para>
-        The previous paragraph is a simplistic explanation of how the build process 
-        uses checksums and shared state memory cache to avoide building tasks that 
-        don't need built.
-        If you want a bit more explanation on the topic, 
-        see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
-        State - What does it mean and why should I care?</ulink>" from the Yocto 
-        Project discussion archives.
-    </para>
-
-    <para>
-        As with all schemes, this one has some drawbacks.
-        It is possible that you could make implicit changes that are not factored into the checksum 
-        calculation, but do affect a task's output. 
-        A good example is perhaps when a tool changes its output.
-        Let's say that the output of <filename>rpmdeps</filename> needed to change.
-        The result of the change should be that all the "package", "package_write_rpm",
-        and "package_deploy-rpm" sstate-cache items would become invalid.
-        But, because this is a change that is external to the code and therefore implicit,
-        the associated sstate-cache items do not become invalidated.
-        In this case, the build process would use the cache items rather than running the 
-        task again. 
-        Obviously, these types of implicit changes can cause problems.
-    </para>
-
-    <para>
-        To avoid these problems during the build, you need to understand the effects of any
-        change you make.
-        Note that any changes you make directly to a function automatically are factored into 
-        the checksum calculation and thus, will invalidate the associated area of sstate cache.
-        You need to be aware of any implicit changes that are not obvious changes to the 
-        code and could affect the output of a given task. 
-        Once you are aware of such a change, you can take steps to invalidate the cache 
-        and force the task to run. 
-        The step to take is as simple as changing a function's comments in the source code. 
-        For example, to invalidate package sstate files, change the comment statments
-        of <filename>do_package</filename> or one of the functions it calls.
-        The change is purely cosmetic, but it causes the checksum to be recalculated and  
-        forces the task to be run again.
-    </para>
-
-    <note>
-        For an example of a commit that makes a cosmetic change to invalidate an sstate, 
-        see this
-        <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
-    </note>
-</section>
--->
-
 </chapter>
 <!-- 
 vim: expandtab tw=80 ts=4