diff options
| author | Scott Rifenbark <scott.m.rifenbark@intel.com> | 2013-01-15 16:29:17 -0800 |
|---|---|---|
| committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2013-01-27 13:54:09 +0000 |
| commit | 89dbdec9929b744303f8653b3c19450a2f7d6c4f (patch) | |
| tree | 624f3e6534df209d03f83c2674704769c4f64c95 /documentation/profile-manual | |
| parent | cdacd8764b26b945253c7a94941806ce34a44da6 (diff) | |
| download | ast2050-yocto-poky-89dbdec9929b744303f8653b3c19450a2f7d6c4f.zip ast2050-yocto-poky-89dbdec9929b744303f8653b3c19450a2f7d6c4f.tar.gz | |
profile-manual: Adding raw text.
Started to put in the usage chapter. Adding text and
sectioning off by paragraph and formatting for literal
sections.
(From yocto-docs rev: c2d328d19023cd6fac6dcaeb8449e2203b5a17b7)
Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/profile-manual')
| -rw-r--r-- | documentation/profile-manual/profile-manual-usage.xml | 1950 |
1 files changed, 862 insertions, 1088 deletions
diff --git a/documentation/profile-manual/profile-manual-usage.xml b/documentation/profile-manual/profile-manual-usage.xml index 65e17e2..71feb41 100644 --- a/documentation/profile-manual/profile-manual-usage.xml +++ b/documentation/profile-manual/profile-manual-usage.xml @@ -2,1215 +2,989 @@ "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > -<chapter id='dev-manual-newbie'> +<chapter id='profile-manual-usage'> -<title>The Yocto Project Open Source Development Environment</title> +<title>Basic Usage (with examples) for each of the Yocto Tracing Tools</title> <para> - This chapter helps you understand the Yocto Project as an open source development project. - In general, working in an open source environment is very different from working in a - closed, proprietary environment. - Additionally, the Yocto Project uses specific tools and constructs as part of its development - environment. - This chapter specifically addresses open source philosophy, licensing issues, code repositories, - the open source distributed version control system Git, and best practices using the Yocto Project. + This chapter presents basic usage examples for each of the tracing + tools. </para> -<section id='open-source-philosophy'> - <title>Open Source Philosophy</title> +<section id='profile-manual-perf'> + <title>perf</title> <para> - Open source philosophy is characterized by software development directed by peer production - and collaboration through an active community of developers. - Contrast this to the more standard centralized development models used by commercial software - companies where a finite set of developers produces a product for sale using a defined set - of procedures that ultimately result in an end product whose architecture and source material - are closed to the public. + The 'perf' tool is the profiling and tracing tool that comes + bundled with the Linux kernel. </para> <para> - Open source projects conceptually have differing concurrent agendas, approaches, and production. - These facets of the development process can come from anyone in the public (community) that has a - stake in the software project. - The open source environment contains new copyright, licensing, domain, and consumer issues - that differ from the more traditional development environment. - In an open source environment, the end product, source material, and documentation are - all available to the public at no cost. + Don't let the fact that it's part of the kernel fool you into thinking + that it's only for tracing and profiling the kernel - you can indeed + use it to trace and profile just the kernel , but you can also use it + to profile specific applications separately (with or without kernel + context), and you can also use it to trace and profile the kernel + and all applications on the system simultaneously to gain a system-wide + view of what's going on. </para> <para> - A benchmark example of an open source project is the Linux Kernel, which was initially conceived - and created by Finnish computer science student Linus Torvalds in 1991. - Conversely, a good example of a non-open source project is the - <trademark class='registered'>Windows</trademark> family of operating - systems developed by <trademark class='registered'>Microsoft</trademark> Corporation. + In many ways, it aims to be a superset of all the tracing and profiling + tools available in Linux today, including all the other tools covered + in this HOWTO. The past couple of years have seen perf subsume a lot + of the functionality of those other tools, and at the same time those + other tools have removed large portions of their previous functionality + and replaced it with calls to the equivalent functionality now + implemented by the perf subsystem. Extrapolation suggests that at + some point those other tools will simply become completely redundant + and go away; until then, we'll cover those other tools in these pages + and in many cases show how the same things can be accomplished in + perf and the other tools when it seems useful to do so. </para> <para> - Wikipedia has a good historical description of the Open Source Philosophy - <ulink url='http://en.wikipedia.org/wiki/Open_source'>here</ulink>. - You can also find helpful information on how to participate in the Linux Community - <ulink url='http://ldn.linuxfoundation.org/book/how-participate-linux-community'>here</ulink>. - </para> -</section> - -<section id="usingpoky-changes-collaborate"> - <title>Using the Yocto Project in a Team Environment</title> - - <para> - It might not be immediately clear how you can use the Yocto Project in a team environment, - or scale it for a large team of developers. - The specifics of any situation determine the best solution. - Granted that the Yocto Project offers immense flexibility regarding this, practices do exist - that experience has shown work well. + The coverage below details some of the most common ways you'll likely + want to apply the tool; full documentation can be found either within + the tool itself or in the man pages at + <ulink url='http://linux.die.net/man/1/perf'>perf(1)</ulink>. </para> - <para> - The core component of any development effort with the Yocto Project is often an - automated build and testing framework along with an image generation process. - You can use these core components to check that the metadata can be built, - highlight when commits break the build, and provide up-to-date images that - allow developers to test the end result and use it as a base platform for further - development. - Experience shows that buildbot is a good fit for this role. - What works well is to configure buildbot to make two types of builds: - incremental and full (from scratch). - See "<ulink url='http://autobuilder.yoctoproject.org:8010/'>Welcome to the buildbot for the Yocto Project</ulink>" - for an example implementation that uses buildbot. - </para> + <section id='perf-setup'> + <title>Setup</title> - <para> - You can tie an incremental build to a commit hook that triggers the build - each time a commit is made to the metadata. - This practice results in useful acid tests that determine whether a given commit - breaks the build in some serious way. - Associating a build to a commit can catch a lot of simple errors. - Furthermore, the tests are fast so developers can get quick feedback on changes. - </para> - - <para> - Full builds build and test everything from the ground up. - These types of builds usually happen at predetermined times like during the - night when the machine load is low. - </para> - - <para> - Most teams have many pieces of software undergoing active development at any given time. - You can derive large benefits by putting these pieces under the control of a source - control system that is compatible (i.e. Git or Subversion (SVN)) with the OpenEmbedded - build system that the Yocto Project uses. - You can then set the autobuilder to pull the latest revisions of the packages - and test the latest commits by the builds. - This practice quickly highlights issues. - The build system easily supports testing configurations that use both a - stable known good revision and a floating revision. - The build system can also take just the changes from specific source control branches. - This capability allows you to track and test specific changes. - </para> - - <para> - Perhaps the hardest part of setting this up is defining the software project or - the metadata policies that surround the different source control systems. - Of course circumstances will be different in each case. - However, this situation reveals one of the Yocto Project's advantages - - the system itself does not - force any particular policy on users, unlike a lot of build systems. - The system allows the best policies to be chosen for the given circumstances. - </para> - - <para> - In general, best practices exist that make your work with the Yocto - Project easier in a team environment. - This list presents some of these practices you might consider following. - Of course, you need to understand that you do not have to follow these - practices and your setup can be totally controlled and customized by - your team: - <itemizedlist> - <listitem><para>Use <link linkend='git'>Git</link> - as the source control system.</para></listitem> - <listitem><para>Maintain your metadata in layers that make sense - for your situation. - See the "<link linkend='understanding-and-creating-layers'>Understanding - and Creating Layers</link>" section for more information on - layers.</para></listitem> - <listitem><para>Separate the project's metadata and code by using - separate Git repositories. - See the "<link linkend='yocto-project-repositories'>Yocto Project - Source Repositories</link>" section for information on these - repositories. - See the "<link linkend='getting-setup'>Getting Set Up</link>" section - for information on how to set up various Yocto Project related - Git repositories.</para></listitem> - <listitem><para>Set up the directory for the shared state cache - (<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>) - where they make sense. - For example, set up the sstate cache for developers using the - same office and share source directories on the developer's - machines.</para></listitem> - <listitem><para>Set up an autobuilder and have it populate the - sstate cache and source directories.</para></listitem> - </itemizedlist> - </para> -</section> + <para> + For this section, we'll assume you've already performed the basic + setup outlined in the General Setup section. + </para> -<section id='yocto-project-repositories'> - <title>Yocto Project Source Repositories</title> + <para> + In particular, you'll get the most mileage out of perf if you + profile an image built with INHIBIT_PACKAGE_STRIP = "1" in your + local.conf. + </para> - <para> - The Yocto Project team maintains complete source repositories for all Yocto Project files - at <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'></ulink>. - This web-based source code browser is organized into categories by function such as - IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and so forth. - From the interface, you can click on any particular item in the "Name" column and - see the URL at the bottom of the page that you need to set up a Git repository for - that particular item. - Having a local Git repository of the Source Directory (poky) allows you to - make changes, contribute to the history, and ultimately enhance the Yocto Project's - tools, Board Support Packages, and so forth. - </para> + <para> + perf runs on the target system for the most part. You can archive + profile data and copy it to the host for analysis, but for the + rest of this document we assume you've ssh'ed to the host and + will be running the perf commands on the target. + </para> + </section> - <para> - Conversely, if you are a developer that is not interested in contributing back to the - Yocto Project, you have the ability to simply download and extract release tarballs - and use them within the Yocto Project environment. - All that is required is a particular release of the Yocto Project and - your application source code. - </para> + <section id='perf-basic-usage'> + <title>Basic Usage</title> - <para> - For any supported release of Yocto Project, you can go to the Yocto Project website’s - <ulink url='&YOCTO_HOME_URL;/download'>download page</ulink> and get a - tarball of the release. - You can also go to this site to download any supported BSP tarballs. - Unpacking the tarball gives you a hierarchical Source Directory that lets you develop - using the Yocto Project. + <para> + The perf tool is pretty much self-documenting. To remind yourself + of the available commands, simply type 'perf', which will show you + basic usage along with the available perf subcommands: + <literallayout class='monospaced'> + root@crownbay:~# perf + + usage: perf [--version] [--help] COMMAND [ARGS] + + The most commonly used perf commands are: + annotate Read perf.data (created by perf record) and display annotated code + archive Create archive with object files with build-ids found in perf.data file + bench General framework for benchmark suites + buildid-cache Manage build-id cache. + buildid-list List the buildids in a perf.data file + diff Read two perf.data files and display the differential profile + evlist List the event names in a perf.data file + inject Filter to augment the events stream with additional information + kmem Tool to trace/measure kernel memory(slab) properties + kvm Tool to trace/measure kvm guest os + list List all symbolic event types + lock Analyze lock events + probe Define new dynamic tracepoints + record Run a command and record its profile into perf.data + report Read perf.data (created by perf record) and display the profile + sched Tool to trace/measure scheduler properties (latencies) + script Read perf.data (created by perf record) and display trace output + stat Run a command and gather performance counter statistics + test Runs sanity tests. + timechart Tool to visualize total system behavior during a workload + top System profiling tool. + + See 'perf help COMMAND' for more information on a specific command. + </literallayout> </para> - <para> - Once you are set up through either tarball extraction or a checkout of Git repositories, - you are ready to develop. - </para> + <section id='using-perf-to-do-basic-profiling'> + <title>Using perf to do Basic Profiling</title> - <para> - In summary, here is where you can get the project files needed for development: - <itemizedlist> - <listitem><para id='source-repositories'><emphasis><ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'>Source Repositories:</ulink></emphasis> - This area contains IDE Plugins, Matchbox, Poky, Poky Support, Tools, Yocto Linux Kernel, and Yocto - Metadata Layers. - You can create local copies of Git repositories for each of these areas.</para> - <para> - <imagedata fileref="figures/source-repos.png" align="center" width="6in" depth="4in" /> - </para></listitem> - <listitem><para><anchor id='index-downloads' /><emphasis><ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink></emphasis> - This area contains index releases such as - the <trademark class='trade'>Eclipse</trademark> - Yocto Plug-in, miscellaneous support, poky, pseudo, installers for cross-development toolchains, - and all released versions of Yocto Project in the form of images or tarballs. - Downloading and extracting these files does not produce a local copy of the - Git repository but rather a snapshot of a particular release or image.</para> - <para> - <imagedata fileref="figures/index-downloads.png" align="center" width="6in" depth="4in" /> - </para></listitem> - <listitem><para><emphasis><ulink url='&YOCTO_HOME_URL;/download'>Yocto Project Download Page</ulink></emphasis> - This page on the Yocto Project website allows you to download any Yocto Project - release or Board Support Package (BSP) in tarball form. - The tarballs are similar to those found in the - <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink> area.</para> - <para> - <imagedata fileref="figures/yp-download.png" align="center" width="6in" depth="4in" /> - </para></listitem> - </itemizedlist> - </para> -</section> + <para> + As a simple test case, we'll profile the 'wget' of a fairly large + file, which is a minimally interesting case because it has both + file and network I/O aspects, and at least in the case of standard + Yocto images, it's implemented as part of busybox, so the methods + we use to analyze it can be used in a very similar way to the whole + host of supported busybox applets in Yocto. + <literallayout class='monospaced'> + root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \ + wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + </literallayout> + The quickest and easiest way to get some basic overall data about + what's going on for a particular workload it to profile it using + 'perf stat'. 'perf stat' basically profiles using a few default + counters and displays the summed counts at the end of the run: + <literallayout class='monospaced'> + root@crownbay:~# perf stat wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + Connecting to downloads.yoctoproject.org (140.211.169.59:80) + linux-2.6.19.2.tar.b 100% |***************************************************| 41727k 0:00:00 ETA + + Performance counter stats for 'wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2': + + 4597.223902 task-clock # 0.077 CPUs utilized + 23568 context-switches # 0.005 M/sec + 68 CPU-migrations # 0.015 K/sec + 241 page-faults # 0.052 K/sec + 3045817293 cycles # 0.663 GHz + <not supported> stalled-cycles-frontend + <not supported> stalled-cycles-backend + 858909167 instructions # 0.28 insns per cycle + 165441165 branches # 35.987 M/sec + 19550329 branch-misses # 11.82% of all branches + + 59.836627620 seconds time elapsed + </literallayout> + Many times such a simple-minded test doesn't yield much of + interest, but sometimes it does (see Real-world Yocto bug + (slow loop-mounted write speed)). + </para> -<section id='yocto-project-terms'> - <title>Yocto Project Terms</title> + <para> + Also, note that 'perf stat' isn't restricted to a fixed set of + counters - basically any event listed in the output of 'perf list' + can be tallied by 'perf stat'. For example, suppose we wanted to + see a summary of all the events related to kernel memory + allocation/freeing along with cache hits and misses: + <literallayout class='monospaced'> + root@crownbay:~# perf stat -e kmem:* -e cache-references -e cache-misses wget http:// downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + Connecting to downloads.yoctoproject.org (140.211.169.59:80) + linux-2.6.19.2.tar.b 100% |***************************************************| 41727k 0:00:00 ETA + + Performance counter stats for 'wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2': + + 5566 kmem:kmalloc + 125517 kmem:kmem_cache_alloc + 0 kmem:kmalloc_node + 0 kmem:kmem_cache_alloc_node + 34401 kmem:kfree + 69920 kmem:kmem_cache_free + 133 kmem:mm_page_free + 41 kmem:mm_page_free_batched + 11502 kmem:mm_page_alloc + 11375 kmem:mm_page_alloc_zone_locked + 0 kmem:mm_page_pcpu_drain + 0 kmem:mm_page_alloc_extfrag + 66848602 cache-references + 2917740 cache-misses # 4.365 % of all cache refs + + 44.831023415 seconds time elapsed + </literallayout> + So 'perf stat' gives us a nice easy way to get a quick overview of + what might be happening for a set of events, but normally we'd + need a little more detail in order to understand what's going on + in a way that we can act on in a useful way. + </para> - <para> - Following is a list of terms and definitions users new to the Yocto Project development - environment might find helpful. - While some of these terms are universal, the list includes them just in case: - <itemizedlist> - <listitem><para><emphasis>Append Files:</emphasis> Files that append build information to - a recipe file. - Append files are known as BitBake append files and <filename>.bbappend</filename> files. - The OpenEmbedded build system expects every append file to have a corresponding and - underlying recipe (<filename>.bb</filename>) file. - Furthermore, the append file and the underlying recipe must have the same root filename. - The filenames can differ only in the file type suffix used (e.g. - <filename>formfactor_0.0.bb</filename> and <filename>formfactor_0.0.bbappend</filename>). - </para> - <para>Information in append files overrides the information in the similarly-named recipe file. - For an example of an append file in use, see the - "<link linkend='using-bbappend-files'>Using .bbappend Files</link>" section. - </para></listitem> - <listitem><para id='bitbake-term'><emphasis>BitBake:</emphasis> - The task executor and scheduler used by - the OpenEmbedded build system to build images. - For more information on BitBake, see the BitBake documentation - in the <filename>bitbake/doc/manual</filename> directory of the - <link linkend='source-directory'>Source Directory</link>.</para></listitem> - <listitem> - <para id='build-directory'><emphasis>Build Directory:</emphasis> - This term refers to the area used by the OpenEmbedded build system for builds. - The area is created when you <filename>source</filename> the setup - environment script that is found in the Source Directory - (i.e. <filename>&OE_INIT_FILE;</filename>). - The <ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink> - variable points to the Build Directory.</para> - - <para>You have a lot of flexibility when creating the Build Directory. - Following are some examples that show how to create the directory: - <itemizedlist> - <listitem><para>Create the Build Directory in your current working directory - and name it <filename>build</filename>. - This is the default behavior. - <literallayout class='monospaced'> - $ source &OE_INIT_PATH; - </literallayout></para></listitem> - <listitem><para>Provide a directory path and specifically name the build - directory. - This next example creates a Build Directory named <filename>YP-&POKYVERSION;</filename> - in your home directory within the directory <filename>mybuilds</filename>. - If <filename>mybuilds</filename> does not exist, the directory is created for you: - <literallayout class='monospaced'> - $ source &OE_INIT_PATH; $HOME/mybuilds/YP-&POKYVERSION; - </literallayout></para></listitem> - <listitem><para>Provide an existing directory to use as the Build Directory. - This example uses the existing <filename>mybuilds</filename> directory - as the Build Directory. - <literallayout class='monospaced'> - $ source &OE_INIT_PATH; $HOME/mybuilds/ - </literallayout></para></listitem> - </itemizedlist> - </para></listitem> - <listitem><para><emphasis>Build System:</emphasis> In the context of the Yocto Project - this term refers to the OpenEmbedded build system used by the project. - This build system is based on the project known as "Poky." - For some historical information about Poky, see the - <link linkend='poky'>Poky</link> term further along in this section. - </para></listitem> - <listitem><para><emphasis>Classes:</emphasis> Files that provide for logic encapsulation - and inheritance allowing commonly used patterns to be defined once and easily used - in multiple recipes. - Class files end with the <filename>.bbclass</filename> filename extension. - </para></listitem> - <listitem><para><emphasis>Configuration File:</emphasis> Configuration information in various - <filename>.conf</filename> files provides global definitions of variables. - The <filename>conf/local.conf</filename> configuration file in the - <link linkend='build-directory'>Build Directory</link> - contains user-defined variables that affect each build. - The <filename>meta-yocto/conf/distro/poky.conf</filename> configuration file - defines Yocto ‘distro’ configuration - variables used only when building with this policy. - Machine configuration files, which - are located throughout the - <link linkend='source-directory'>Source Directory</link>, define - variables for specific hardware and are only used when building for that target - (e.g. the <filename>machine/beagleboard.conf</filename> configuration file defines - variables for the Texas Instruments ARM Cortex-A8 development board). - Configuration files end with a <filename>.conf</filename> filename extension. - </para></listitem> - <listitem><para><emphasis>Cross-Development Toolchain:</emphasis> - A collection of software development - tools and utilities that allow you to develop software for targeted architectures. - This toolchain contains cross-compilers, linkers, and debuggers that are specific to - an architecture. - You can use the OpenEmbedded build system to build a cross-development toolchain - installer that when run installs the toolchain that contains the development tools you - need to cross-compile and test your software. - The Yocto Project ships with images that contain installers for - toolchains for supported architectures as well. - Sometimes this toolchain is referred to as the meta-toolchain.</para></listitem> - <listitem><para><emphasis>Image:</emphasis> An image is the result produced when - BitBake processes a given collection of recipes and related metadata. - Images are the binary output that run on specific hardware or QEMU - and for specific use cases. - For a list of the supported image types that the Yocto Project provides, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" - chapter in the Yocto Project Reference Manual.</para></listitem> - <listitem><para id='layer'><emphasis>Layer:</emphasis> A collection of recipes representing the core, - a BSP, or an application stack. - For a discussion on BSP Layers, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" - section in the Yocto Project Board Support Packages (BSP) Developer's Guide.</para></listitem> - <listitem><para id='metadata'><emphasis>Metadata:</emphasis> The files that BitBake parses when - building an image. - Metadata includes recipes, classes, and configuration files.</para></listitem> - <listitem><para id='oe-core'><emphasis>OE-Core:</emphasis> A core set of metadata originating - with OpenEmbedded (OE) that is shared between OE and the Yocto Project. - This metadata is found in the <filename>meta</filename> directory of the source - directory.</para></listitem> - <listitem><para><emphasis>Package:</emphasis> In the context of the Yocto Project, - this term refers to the packaged output from a baked recipe. - A package is generally the compiled binaries produced from the recipe's sources. - You ‘bake’ something by running it through BitBake.</para> - <para>It is worth noting that the term "package" can, in general, have subtle - meanings. For example, the packages refered to in the - "<ulink url='&YOCTO_DOCS_QS_URL;#packages'>The Packages</ulink>" section are - compiled binaries that when installed add functionality to your Linux - distribution.</para> - <para>Another point worth noting is that historically within the Yocto Project, - recipes were referred to as packages - thus, the existence of several BitBake - variables that are seemingly mis-named, - (e.g. <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-PRINC'><filename>PRINC</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>, and - <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>). - </para></listitem> - <listitem><para id='poky'><emphasis>Poky:</emphasis> The term "poky" can mean several things. - In its most general sense, it is an open-source project that was initially developed - by OpenedHand. With OpenedHand, poky was developed off of the existing OpenEmbedded - build system becoming a build system for embedded images. - After Intel Corporation acquired OpenedHand, the project poky became the basis for - the Yocto Project's build system. - Within the Yocto Project source repositories, poky exists as a separate Git repository - that can be cloned to yield a local copy on the host system. - Thus, "poky" can refer to the local copy of the Source Directory used to develop within - the Yocto Project.</para></listitem> - <listitem><para><emphasis>Recipe:</emphasis> A set of instructions for building packages. - A recipe describes where you get source code and which patches to apply. - Recipes describe dependencies for libraries or for other recipes, and they - also contain configuration and compilation options. - Recipes contain the logical unit of execution, the software/images to build, and - use the <filename>.bb</filename> file extension.</para></listitem> - <listitem> - <para id='source-directory'><emphasis>Source Directory:</emphasis> - This term refers to the directory structure created as a result of either downloading - and unpacking a Yocto Project release tarball or creating a local copy of - the <filename>poky</filename> Git repository - <filename>git://git.yoctoproject.org/poky</filename>. - Sometimes you might hear the term "poky directory" used to refer to this - directory structure. - <note> - The OpenEmbedded build system does not support file or directory names that - contain spaces. - Be sure that the Source Directory you use does not contain these types - of names. - </note></para> - <para>The Source Directory contains BitBake, Documentation, metadata and - other files that all support the Yocto Project. - Consequently, you must have the Source Directory in place on your development - system in order to do any development using the Yocto Project.</para> - - <para>For tarball expansion, the name of the top-level directory of the Source Directory - is derived from the Yocto Project release tarball. - For example, downloading and unpacking <filename>&YOCTO_POKY_TARBALL;</filename> - results in a Source Directory whose top-level folder is named - <filename>&YOCTO_POKY;</filename>. - If you create a local copy of the Git repository, then you can name the repository - anything you like. - Throughout much of the documentation, <filename>poky</filename> is used as the name of - the top-level folder of the local copy of the poky Git repository. - So, for example, cloning the <filename>poky</filename> Git repository results in a - local Git repository whose top-level folder is also named <filename>poky</filename>.</para> - - <para>It is important to understand the differences between the Source Directory created - by unpacking a released tarball as compared to cloning - <filename>git://git.yoctoproject.org/poky</filename>. - When you unpack a tarball, you have an exact copy of the files based on the time of - release - a fixed release point. - Any changes you make to your local files in the Source Directory are on top of the release. - On the other hand, when you clone the <filename>poky</filename> Git repository, you have an - active development repository. - In this case, any local changes you make to the Source Directory can be later applied - to active development branches of the upstream <filename>poky</filename> Git - repository.</para> - - <para>Finally, if you want to track a set of local changes while starting from the same point - as a release tarball, you can create a local Git branch that - reflects the exact copy of the files at the time of their release. - You do this by using Git tags that are part of the repository.</para> - - <para>For more information on concepts related to Git repositories, branches, and tags, - see the - "<link linkend='repositories-tags-and-branches'>Repositories, Tags, and Branches</link>" - section.</para></listitem> - <listitem><para><emphasis>Tasks:</emphasis> Arbitrary groups of software Recipes. - You simply use Tasks to hold recipes that, when built, usually accomplish a single task. - For example, a task could contain the recipes for a company’s proprietary or value-add software. - Or, the task could contain the recipes that enable graphics. - A task is really just another recipe. - Because task files are recipes, they end with the <filename>.bb</filename> filename - extension.</para></listitem> - <listitem><para><emphasis>Upstream:</emphasis> A reference to source code or repositories - that are not local to the development system but located in a master area that is controlled - by the maintainer of the source code. - For example, in order for a developer to work on a particular piece of code, they need to - first get a copy of it from an "upstream" source.</para></listitem> - </itemizedlist> - </para> -</section> + <para> + To dive down into a next level of detail, we can use 'perf + record'/'perf report' which will collect profiling data and + present it to use using an interactive text-based UI (or + simply as text if we specify --stdio to 'perf report'). + </para> -<section id='licensing'> - <title>Licensing</title> + <para> + As our first attempt at profiling this workload, we'll simply + run 'perf record', handing it the workload we want to profile + (everything after 'perf record' and any perf options we hand + it - here none - will be executedin a new shell). perf collects + samples until the process exits and records them in a file named + 'perf.data' in the current working directory. + <literallayout class='monospaced'> + root@crownbay:~# perf record wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 - <para> - Because open source projects are open to the public, they have different licensing structures in place. - License evolution for both Open Source and Free Software has an interesting history. - If you are interested in this history, you can find basic information here: - <itemizedlist> - <listitem><para><ulink url='http://en.wikipedia.org/wiki/Open-source_license'>Open source license history</ulink> - </para></listitem> - <listitem><para><ulink url='http://en.wikipedia.org/wiki/Free_software_license'>Free software license - history</ulink></para></listitem> - </itemizedlist> - </para> + Connecting to downloads.yoctoproject.org (140.211.169.59:80) + linux-2.6.19.2.tar.b 100% |************************************************| 41727k 0:00:00 ETA + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ] + </literallayout> + To see the results in a 'text-based UI' (tui), simply run + 'perf report', which will read the perf.data file in the current + working directory and display the results in an interactive UI: + <literallayout class='monospaced'> + root@crownbay:~# perf report + </literallayout> + </para> - <para> - In general, the Yocto Project is broadly licensed under the Massachusetts Institute of Technology - (MIT) License. - MIT licensing permits the reuse of software within proprietary software as long as the - license is distributed with that software. - MIT is also compatible with the GNU General Public License (GPL). - Patches to the Yocto Project follow the upstream licensing scheme. - You can find information on the MIT license at - <ulink url='http://www.opensource.org/licenses/mit-license.php'>here</ulink>. - You can find information on the GNU GPL <ulink url='http://www.opensource.org/licenses/LGPL-3.0'> - here</ulink>. - </para> + <para> + <imagedata fileref="figures/perf-wget-flat-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - When you build an image using the Yocto Project, the build process uses a - known list of licenses to ensure compliance. - You can find this list in the Yocto Project files directory at - <filename>meta/files/common-licenses</filename>. - Once the build completes, the list of all licenses found and used during that build are - kept in the - <link linkend='build-directory'>Build Directory</link> at - <filename>tmp/deploy/images/licenses</filename>. - </para> + <para> + The above screenshot displays a 'flat' profile, one entry for + each 'bucket' corresponding to the functions that were profiled + during the profiling run, ordered from the most popular to the + least (perf has options to sort in various orders and keys as + well as display entries only above a certain threshold and so + on - see the perf documentation for details). Note that this + includes both userspace functions (entries containing a [.]) and + kernel functions accounted to the process (entries containing + a [k]). (perf has command-line modifiers that can be used to + restrict the profiling to kernel or userspace, among others). + </para> - <para> - If a module requires a license that is not in the base list, the build process - generates a warning during the build. - These tools make it easier for a developer to be certain of the licenses with which - their shipped products must comply. - However, even with these tools it is still up to the developer to resolve potential licensing issues. - </para> + <para> + Notice also that the above report shows an entry for 'busybox', + which is the executable that implements 'wget' in Yocto, but that + instead of a useful function name in that entry, it displays + an not-so-friendly hex value instead. The steps below will show + how to fix that problem. + </para> - <para> - The base list of licenses used by the build process is a combination of the Software Package - Data Exchange (SPDX) list and the Open Source Initiative (OSI) projects. - <ulink url='http://spdx.org'>SPDX Group</ulink> is a working group of the Linux Foundation - that maintains a specification - for a standard format for communicating the components, licenses, and copyrights - associated with a software package. - <ulink url='http://opensource.org'>OSI</ulink> is a corporation dedicated to the Open Source - Definition and the effort for reviewing and approving licenses that are OSD-conformant. - </para> + <para> + Before we do that, however, let's try running a different profile, + one which shows something a little more interesting. The only + difference between the new profile and the previous one is that + we'll add the -g option, which will record not just the address + of a sampled function, but the entire callchain to the sampled + function as well: + <literallayout class='monospaced'> + root@crownbay:~# perf record -g wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + Connecting to downloads.yoctoproject.org (140.211.169.59:80) + linux-2.6.19.2.tar.b 100% |************************************************| 41727k 0:00:00 ETA + [ perf record: Woken up 3 times to write data ] + [ perf record: Captured and wrote 0.652 MB perf.data (~28476 samples) ] - <para> - You can find a list of the combined SPDX and OSI licenses that the Yocto Project uses - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/files/common-licenses'>here</ulink>. - This wiki page discusses the license infrastructure used by the Yocto Project. - </para> - <para> - For information that can help you to maintain compliance with various open source licensing - during the lifecycle of a product created using the Yocto Project, see the - "<link linkend='maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Product's Lifecycle</link>" section. - </para> -</section> + root@crownbay:~# perf report + </literallayout> + </para> -<section id='git'> - <title>Git</title> + <para> + <imagedata fileref="figures/perf-wget-g-copy-to-user-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - The Yocto Project uses Git, which is a free, open source distributed version control system. - Git supports distributed development, non-linear development, and can handle large projects. - It is best that you have some fundamental understanding of how Git tracks projects and - how to work with Git if you are going to use Yocto Project for development. - This section provides a quick overview of how Git works and provides you with a summary - of some essential Git commands. - </para> + <para> + Using the callgraph view, we can actually see not only which + functions took the most time, but we can also see a summary of + how those functions were called and learn something about how the + program interacts with the kernel in the process. + </para> - <para> - For more information on Git, see - <ulink url='http://git-scm.com/documentation'></ulink>. - If you need to download Git, go to <ulink url='http://git-scm.com/download'></ulink>. - </para> + <para> + Notice that each entry in the above screenshot now contains a '+' + on the left-hand side. This means that we can expand the entry and + drill down into the callchains that feed into that entry. + Pressing 'enter' on any one of them will expand the callchain + (you can also press 'E' to expand them all at the same time or 'C' + to collapse them all). + </para> - <section id='repositories-tags-and-branches'> - <title>Repositories, Tags, and Branches</title> + <para> + In the screenshot above, we've toggled the __copy_to_user_ll() + entry and several subnodes all the way down. This lets us see + which callchains contributed to the profiled __copy_to_user_ll() + function which contributed 1.77% to the total profile. + </para> <para> - As mentioned earlier in section - "<link linkend='yocto-project-repositories'>Yocto Project Source Repositories</link>", - the Yocto Project maintains source repositories at - <ulink url='&YOCTO_GIT_URL;/cgit.cgi'></ulink>. - If you look at this web-interface of the repositories, each item is a separate - Git repository. + As a bit of background explanation for these callchains, think + about what happens at a high level when you run wget to get a file + out on the network. Basically what happens is that the data comes + into the kernel via the network connection (socket) and is passed + to the userspace program 'wget' (which is actually a part of + busybox, but that's not important for now), which takes the buffers + the kernel passes to it and writes it to a disk file to save it. </para> <para> - Git repositories use branching techniques that track content change (not files) - within a project (e.g. a new feature or updated documentation). - Creating a tree-like structure based on project divergence allows for excellent historical - information over the life of a project. - This methodology also allows for an environment in which you can do lots of - local experimentation on a project as you develop changes or new features. + The part of this process that we're looking at in the above call + stacks is the part where the kernel passes the data it's read from + the socket down to wget i.e. a copy-to-user. </para> <para> - A Git repository represents all development efforts for a given project. - For example, the Git repository <filename>poky</filename> contains all changes - and developments for Poky over the course of its entire life. - That means that all changes that make up all releases are captured. - The repository maintains a complete history of changes. + Notice also that here there's also a case where the a hex value + is displayed in the callstack, here in the expanded + sys_clock_gettime() function. Later we'll see it resolve to a + userspace function call in busybox. </para> <para> - You can create a local copy of any repository by "cloning" it with the Git - <filename>clone</filename> command. - When you clone a Git repository, you end up with an identical copy of the - repository on your development system. - Once you have a local copy of a repository, you can take steps to develop locally. - For examples on how to clone Git repositories, see the section - "<link linkend='getting-setup'>Getting Set Up</link>" earlier in this manual. + <imagedata fileref="figures/perf-wget-g-copy-from-user-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> </para> <para> - It is important to understand that Git tracks content change and not files. - Git uses "branches" to organize different development efforts. - For example, the <filename>poky</filename> repository has - <filename>bernard</filename>, - <filename>edison</filename>, <filename>denzil</filename>, <filename>danny</filename> - and <filename>master</filename> branches among others. - You can see all the branches by going to - <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and - clicking on the - <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/heads'>[...]</ulink></filename> - link beneath the "Branch" heading. + The above screenshot shows the other half of the journey for the + data - from the wget program's userspace buffers to disk. To get + the buffers to disk, the wget program issues a write(2), which + does a copy-from-user to the kernel, which then takes care via + some circuitous path (probably also present somewhere in the + profile data), to get it safely to disk. </para> <para> - Each of these branches represents a specific area of development. - The <filename>master</filename> branch represents the current or most recent - development. - All other branches represent off-shoots of the <filename>master</filename> - branch. + Now that we've seen the basic layout of the profile data and the + basics of how to extract useful information out of it, let's get + back to the task at hand and see if we can get some basic idea + about where the time is spent in the program we're profiling, + wget. Remember that wget is actually implemented as an applet + in busybox, so while the process name is 'wget', the executable + we're actually interested in is busybox. So let's expand the + first entry containing busybox: </para> <para> - When you create a local copy of a Git repository, the copy has the same set - of branches as the original. - This means you can use Git to create a local working area (also called a branch) - that tracks a specific development branch from the source Git repository. - in other words, you can define your local Git environment to work on any development - branch in the repository. - To help illustrate, here is a set of commands that creates a local copy of the - <filename>poky</filename> Git repository and then creates and checks out a local - Git branch that tracks the Yocto Project &DISTRO; Release (&DISTRO_NAME;) development: - <literallayout class='monospaced'> - $ cd ~ - $ git clone git://git.yoctoproject.org/poky - $ cd poky - $ git checkout -b &DISTRO_NAME; origin/&DISTRO_NAME; - </literallayout> - In this example, the name of the top-level directory of your local Yocto Project - Files Git repository is <filename>poky</filename>, - and the name of the local working area (or local branch) you have created and checked - out is <filename>&DISTRO_NAME;</filename>. - The files in your repository now reflect the same files that are in the - <filename>&DISTRO_NAME;</filename> development branch of the Yocto Project's - <filename>poky</filename> repository. - It is important to understand that when you create and checkout a - local working branch based on a branch name, - your local environment matches the "tip" of that development branch - at the time you created your local branch, which could be - different than the files at the time of a similarly named release. - In other words, creating and checking out a local branch based on the - <filename>&DISTRO_NAME;</filename> branch name is not the same as - cloning and checking out the <filename>master</filename> branch. - Keep reading to see how you create a local snapshot of a Yocto Project Release. + <imagedata fileref="figures/perf-wget-busybox-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> </para> <para> - Git uses "tags" to mark specific changes in a repository. - Typically, a tag is used to mark a special point such as the final change - before a project is released. - You can see the tags used with the <filename>poky</filename> Git repository - by going to <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and - clicking on the - <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/tags'>[...]</ulink></filename> - link beneath the "Tag" heading. + Again, before we expanded we saw that the function was labeled + with a hex value instead of a symbol as with most of the kernel + entries. Expanding the busybox entry doesn't make it any better. </para> <para> - Some key tags are <filename>bernard-5.0</filename>, <filename>denzil-7.0</filename>, - and <filename>&DISTRO_NAME;-&POKYVERSION;</filename>. - These tags represent Yocto Project releases. + The problem is that perf can't find the symbol information for the + busybox binary, which is actually stripped out by the Yocto build + system. </para> <para> - When you create a local copy of the Git repository, you also have access to all the - tags. - Similar to branches, you can create and checkout a local working Git branch based - on a tag name. - When you do this, you get a snapshot of the Git repository that reflects - the state of the files when the change was made associated with that tag. - The most common use is to checkout a working branch that matches a specific - Yocto Project release. - Here is an example: + One way around that is to put the following in your local.conf + when you build the image: <literallayout class='monospaced'> - $ cd ~ - $ git clone git://git.yoctoproject.org/poky - $ cd poky - $ git checkout -b my-&DISTRO_NAME;-&POKYVERSION; &DISTRO_NAME;-&POKYVERSION; + INHIBIT_PACKAGE_STRIP = "1" </literallayout> - In this example, the name of the top-level directory of your local Yocto Project - Files Git repository is <filename>poky</filename>. - And, the name of the local branch you have created and checked out is - <filename>my-&DISTRO_NAME;-&POKYVERSION;</filename>. - The files in your repository now exactly match the Yocto Project &DISTRO; - Release tag (<filename>&DISTRO_NAME;-&POKYVERSION;</filename>). - It is important to understand that when you create and checkout a local - working branch based on a tag, your environment matches a specific point - in time and not a development branch. + However, we already have an image with the binaries stripped, + so what can we do to get perf to resolve the symbols? Basically + we need to install the debuginfo for the busybox package. </para> - </section> - <section id='basic-commands'> - <title>Basic Commands</title> + <para> + To generate the debug info for the packages in the image, we can + to add dbg-pkgs to EXTRA_IMAGE_FEATURES in local.conf. For example: + <literallayout class='monospaced'> + EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs" + </literallayout> + Additionally, in order to generate the type of debuginfo that + perf understands, we also need to add the following to local.conf: + <literallayout class='monospaced'> + PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory' + </literallayout> + Once we've done that, we can install the debuginfo for busybox. + The debug packages once built can be found in + build/tmp/deploy/rpm/* on the host system. Find the + busybox-dbg-...rpm file and copy it to the target. For example: + <literallayout class='monospaced'> + [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2/busybox-dbg-1.20.2-r2.core2.rpm root@192.168.1.31: + root@192.168.1.31's password: + busybox-dbg-1.20.2-r2.core2.rpm 100% 1826KB 1.8MB/s 00:01 + </literallayout> + Now install the debug rpm on the target: + <literallayout class='monospaced'> + root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2.rpm + </literallayout> + Now that the debuginfo is installed, we see that the busybox + entries now display their functions symbolically: + </para> <para> - Git has an extensive set of commands that lets you manage changes and perform - collaboration over the life of a project. - Conveniently though, you can manage with a small set of basic operations and workflows - once you understand the basic philosophy behind Git. - You do not have to be an expert in Git to be functional. - A good place to look for instruction on a minimal set of Git commands is - <ulink url='http://git-scm.com/documentation'>here</ulink>. - If you need to download Git, you can do so - <ulink url='http://git-scm.com/download'>here</ulink>. + <imagedata fileref="figures/perf-wget-busybox-debuginfo.png" width="6in" depth="7in" align="center" scalefit="1" /> </para> <para> - If you don’t know much about Git, we suggest you educate - yourself by visiting the links previously mentioned. + If we expand one of the entries and press 'enter' on a leaf node, + we're presented with a menu of actions we can take to get more + information related to that entry: </para> <para> - The following list briefly describes some basic Git operations as a way to get started. - As with any set of commands, this list (in most cases) simply shows the base command and - omits the many arguments they support. - See the Git documentation for complete descriptions and strategies on how to use these commands: - <itemizedlist> - <listitem><para><emphasis><filename>git init</filename>:</emphasis> Initializes an empty Git repository. - You cannot use Git commands unless you have a <filename>.git</filename> repository.</para></listitem> - <listitem><para><emphasis><filename>git clone</filename>:</emphasis> Creates a clone of a repository. - During collaboration, this command allows you to create a local repository that is on - equal footing with a fellow developer’s repository.</para></listitem> - <listitem><para><emphasis><filename>git add</filename>:</emphasis> Adds updated file contents - to the index that - Git uses to track changes. - You must add all files that have changed before you can commit them.</para></listitem> - <listitem><para><emphasis><filename>git commit</filename>:</emphasis> Creates a “commit” that documents - the changes you made. - Commits are used for historical purposes, for determining if a maintainer of a project - will allow the change, and for ultimately pushing the change from your local Git repository - into the project’s upstream (or master) repository.</para></listitem> - <listitem><para><emphasis><filename>git status</filename>:</emphasis> Reports any modified files that - possibly need to be added and committed.</para></listitem> - <listitem><para><emphasis><filename>git checkout <branch-name></filename>:</emphasis> Changes - your working branch. - This command is analogous to “cd”.</para></listitem> - <listitem><para><emphasis><filename>git checkout –b <working-branch></filename>:</emphasis> Creates - a working branch on your local machine where you can isolate work. - It is a good idea to use local branches when adding specific features or changes. - This way if you don’t like what you have done you can easily get rid of the work.</para></listitem> - <listitem><para><emphasis><filename>git branch</filename>:</emphasis> Reports - existing local branches and - tells you the branch in which you are currently working.</para></listitem> - <listitem><para><emphasis><filename>git branch -D <branch-name></filename>:</emphasis> - Deletes an existing local branch. - You need to be in a local branch other than the one you are deleting - in order to delete <filename><branch-name></filename>.</para></listitem> - <listitem><para><emphasis><filename>git pull</filename>:</emphasis> Retrieves information - from an upstream Git - repository and places it in your local Git repository. - You use this command to make sure you are synchronized with the repository - from which you are basing changes (.e.g. the master branch).</para></listitem> - <listitem><para><emphasis><filename>git push</filename>:</emphasis> Sends all your local changes you - have committed to an upstream Git repository (e.g. a contribution repository). - The maintainer of the project draws from these repositories when adding your changes to the - project’s master repository.</para></listitem> - <listitem><para><emphasis><filename>git merge</filename>:</emphasis> Combines or adds changes from one - local branch of your repository with another branch. - When you create a local Git repository, the default branch is named “master”. - A typical workflow is to create a temporary branch for isolated work, make and commit your - changes, switch to your local master branch, merge the changes from the temporary branch into the - local master branch, and then delete the temporary branch.</para></listitem> - <listitem><para><emphasis><filename>git cherry-pick</filename>:</emphasis> Choose and apply specific - commits from one branch into another branch. - There are times when you might not be able to merge all the changes in one branch with - another but need to pick out certain ones.</para></listitem> - <listitem><para><emphasis><filename>gitk</filename>:</emphasis> Provides a GUI view of the branches - and changes in your local Git repository. - This command is a good way to graphically see where things have diverged in your - local repository.</para></listitem> - <listitem><para><emphasis><filename>git log</filename>:</emphasis> Reports a history of your changes to the - repository.</para></listitem> - <listitem><para><emphasis><filename>git diff</filename>:</emphasis> Displays line-by-line differences - between your local working files and the same files in the upstream Git repository that your - branch currently tracks.</para></listitem> - </itemizedlist> + <imagedata fileref="figures/perf-wget-busybox-dso-zoom-menu.png" width="6in" depth="7in" align="center" scalefit="1" /> </para> - </section> -</section> -<section id='workflows'> - <title>Workflows</title> + <para> + One of these actions allows us to show a view that displays a + busybox-centric view of the profiled functions (in this case we've + also expanded all the nodes using the 'E' key): + </para> - <para> - This section provides some overview on workflows using Git. - In particular, the information covers basic practices that describe roles and actions in a - collaborative development environment. - Again, if you are familiar with this type of development environment, you might want to just - skip this section. - </para> + <para> + <imagedata fileref="figures/perf-wget-busybox-dso-zoom.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - The Yocto Project files are maintained using Git in a "master" branch whose Git history - tracks every change and whose structure provides branches for all diverging functionality. - Although there is no need to use Git, many open source projects do so. - For the Yocto Project, a key individual called the "maintainer" is responsible for the "master" - branch of the Git repository. - The "master" branch is the “upstream” repository where the final builds of the project occur. - The maintainer is responsible for allowing changes in from other developers and for - organizing the underlying branch structure to reflect release strategies and so forth. - <note>You can see who is the maintainer for Yocto Project files by examining the - <filename>maintainers.inc</filename> file in the Yocto Project - <filename>meta-yocto/conf/distro/include</filename> directory.</note> - </para> + <para> + Finally, we can see that now that the busybox debuginfo is + installed, the previously unresolved symbol in the + sys_clock_gettime() entry mentioned previously is now resolved, + and shows that the sys_clock_gettime system call that was the + source of 6.75% of the copy-to-user overhead was initiated by + the handle_input() busybox function: + </para> - <para> - The project also has contribution repositories known as “contrib” areas. - These areas temporarily hold changes to the project that have been submitted or committed - by the Yocto Project development team and by community members that contribute to the project. - The maintainer determines if the changes are qualified to be moved from the "contrib" areas - into the "master" branch of the Git repository. - </para> + <para> + <imagedata fileref="figures/perf-wget-g-copy-to-user-expanded-debuginfo.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - Developers (including contributing community members) create and maintain cloned repositories - of the upstream "master" branch. - These repositories are local to their development platforms and are used to develop changes. - When a developer is satisfied with a particular feature or change, they “push” the changes - to the appropriate "contrib" repository. - </para> + <para> + At the lowest level of detail, we can dive down to the assembly + level and see which instructions caused the most overhead in a + function. Pressing 'enter' on the 'udhcpc_main' function, we're + again presented with a menu: + </para> - <para> - Developers are responsible for keeping their local repository up-to-date with "master". - They are also responsible for straightening out any conflicts that might arise within files - that are being worked on simultaneously by more than one person. - All this work is done locally on the developer’s machine before anything is pushed to a - "contrib" area and examined at the maintainer’s level. - </para> + <para> + <imagedata fileref="figures/perf-wget-busybox-annotate-menu.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - A somewhat formal method exists by which developers commit changes and push them into the - "contrib" area and subsequently request that the maintainer include them into "master" - This process is called “submitting a patch” or “submitting a change.” - For information on submitting patches and changes, see the - "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" section. - </para> + <para> + Selecting 'Annotate udhcpc_main', we get a detailed listing of + percentages by instruction for the udhcpc_main function. From the + display, we can see that over 50% of the time spent in this + function is taken up by a couple tests and the move of a + constant (1) to a register: + </para> - <para> - To summarize the environment: we have a single point of entry for changes into the project’s - "master" branch of the Git repository, which is controlled by the project’s maintainer. - And, we have a set of developers who independently develop, test, and submit changes - to "contrib" areas for the maintainer to examine. - The maintainer then chooses which changes are going to become a permanent part of the project. - </para> + <para> + <imagedata fileref="figures/perf-wget-busybox-annotate-udhcpc.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - <imagedata fileref="figures/git-workflow.png" width="6in" depth="3in" align="left" scalefit="1" /> - </para> + <para> + As a segue into tracing, let's try another profile using a + different counter, something other than the default 'cycles'. + </para> - <para> - While each development environment is unique, there are some best practices or methods - that help development run smoothly. - The following list describes some of these practices. - For more information about Git workflows, see the workflow topics in the - <ulink url='http://book.git-scm.com'>Git Community Book</ulink>. - <itemizedlist> - <listitem><para><emphasis>Make Small Changes:</emphasis> It is best to keep the changes you commit - small as compared to bundling many disparate changes into a single commit. - This practice not only keeps things manageable but also allows the maintainer - to more easily include or refuse changes.</para> - <para>It is also good practice to leave the repository in a state that allows you to - still successfully build your project. In other words, do not commit half of a feature, - then add the other half in a separate, later commit. - Each commit should take you from one buildable project state to another - buildable state.</para></listitem> - <listitem><para><emphasis>Use Branches Liberally:</emphasis> It is very easy to create, use, and - delete local branches in your working Git repository. - You can name these branches anything you like. - It is helpful to give them names associated with the particular feature or change - on which you are working. - Once you are done with a feature or change, simply discard the branch.</para></listitem> - <listitem><para><emphasis>Merge Changes:</emphasis> The <filename>git merge</filename> - command allows you to take the - changes from one branch and fold them into another branch. - This process is especially helpful when more than a single developer might be working - on different parts of the same feature. - Merging changes also automatically identifies any collisions or “conflicts” - that might happen as a result of the same lines of code being altered by two different - developers.</para></listitem> - <listitem><para><emphasis>Manage Branches:</emphasis> Because branches are easy to use, you should - use a system where branches indicate varying levels of code readiness. - For example, you can have a “work” branch to develop in, a “test” branch where the code or - change is tested, a “stage” branch where changes are ready to be committed, and so forth. - As your project develops, you can merge code across the branches to reflect ever-increasing - stable states of the development.</para></listitem> - <listitem><para><emphasis>Use Push and Pull:</emphasis> The push-pull workflow is based on the - concept of developers “pushing” local commits to a remote repository, which is - usually a contribution repository. - This workflow is also based on developers “pulling” known states of the project down into their - local development repositories. - The workflow easily allows you to pull changes submitted by other developers from the - upstream repository into your work area ensuring that you have the most recent software - on which to develop. - The Yocto Project has two scripts named <filename>create-pull-request</filename> and - <filename>send-pull-request</filename> that ship with the release to facilitate this - workflow. - You can find these scripts in the local Yocto Project files Git repository in - the <filename>scripts</filename> directory.</para> - <para>You can find more information on these scripts in the - "<link linkend='pushing-a-change-upstream'>Using - Scripts to Push a Change Upstream and Request a Pull</link>" section. - </para></listitem> - <listitem><para><emphasis>Patch Workflow:</emphasis> This workflow allows you to notify the - maintainer through an email that you have a change (or patch) you would like considered - for the "master" branch of the Git repository. - To send this type of change you format the patch and then send the email using the Git commands - <filename>git format-patch</filename> and <filename>git send-email</filename>. - You can find information on how to submit changes - later in this chapter.</para></listitem> - </itemizedlist> - </para> -</section> + <para> + The tracing and profiling infrastructure in Linux has become + unified in a way that allows us to use the same tool with a + completely different set of counters, not just the standard + hardware counters that traditionally tools have had to restrict + themselves to (of course the traditional tools can also make use + of the expanded possibilities now available to them, and in some + cases have, as mentioned previously). + </para> -<section id='tracking-bugs'> - <title>Tracking Bugs</title> + <para> + We can get a list of the available events that can be used to + profile a workload via 'perf list': + <literallayout class='monospaced'> + root@crownbay:~# perf list + + List of pre-defined events (to be used in -e): + cpu-cycles OR cycles [Hardware event] + stalled-cycles-frontend OR idle-cycles-frontend [Hardware event] + stalled-cycles-backend OR idle-cycles-backend [Hardware event] + instructions [Hardware event] + cache-references [Hardware event] + cache-misses [Hardware event] + branch-instructions OR branches [Hardware event] + branch-misses [Hardware event] + bus-cycles [Hardware event] + ref-cycles [Hardware event] + + cpu-clock [Software event] + task-clock [Software event] + page-faults OR faults [Software event] + minor-faults [Software event] + major-faults [Software event] + context-switches OR cs [Software event] + cpu-migrations OR migrations [Software event] + alignment-faults [Software event] + emulation-faults [Software event] + + L1-dcache-loads [Hardware cache event] + L1-dcache-load-misses [Hardware cache event] + L1-dcache-prefetch-misses [Hardware cache event] + L1-icache-loads [Hardware cache event] + L1-icache-load-misses [Hardware cache event] + . + . + . + rNNN [Raw hardware event descriptor] + cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] + (see 'perf list --help' on how to encode it) + + mem:<addr>[:access] [Hardware breakpoint] + + sunrpc:rpc_call_status [Tracepoint event] + sunrpc:rpc_bind_status [Tracepoint event] + sunrpc:rpc_connect_status [Tracepoint event] + sunrpc:rpc_task_begin [Tracepoint event] + skb:kfree_skb [Tracepoint event] + skb:consume_skb [Tracepoint event] + skb:skb_copy_datagram_iovec [Tracepoint event] + net:net_dev_xmit [Tracepoint event] + net:net_dev_queue [Tracepoint event] + net:netif_receive_skb [Tracepoint event] + net:netif_rx [Tracepoint event] + napi:napi_poll [Tracepoint event] + sock:sock_rcvqueue_full [Tracepoint event] + sock:sock_exceed_buf_limit [Tracepoint event] + udp:udp_fail_queue_rcv_skb [Tracepoint event] + hda:hda_send_cmd [Tracepoint event] + hda:hda_get_response [Tracepoint event] + hda:hda_bus_reset [Tracepoint event] + scsi:scsi_dispatch_cmd_start [Tracepoint event] + scsi:scsi_dispatch_cmd_error [Tracepoint event] + scsi:scsi_eh_wakeup [Tracepoint event] + drm:drm_vblank_event [Tracepoint event] + drm:drm_vblank_event_queued [Tracepoint event] + drm:drm_vblank_event_delivered [Tracepoint event] + random:mix_pool_bytes [Tracepoint event] + random:mix_pool_bytes_nolock [Tracepoint event] + random:credit_entropy_bits [Tracepoint event] + gpio:gpio_direction [Tracepoint event] + gpio:gpio_value [Tracepoint event] + block:block_rq_abort [Tracepoint event] + block:block_rq_requeue [Tracepoint event] + block:block_rq_issue [Tracepoint event] + block:block_bio_bounce [Tracepoint event] + block:block_bio_complete [Tracepoint event] + block:block_bio_backmerge [Tracepoint event] + . + . + writeback:writeback_wake_thread [Tracepoint event] + writeback:writeback_wake_forker_thread [Tracepoint event] + writeback:writeback_bdi_register [Tracepoint event] + . + . + writeback:writeback_single_inode_requeue [Tracepoint event] + writeback:writeback_single_inode [Tracepoint event] + kmem:kmalloc [Tracepoint event] + kmem:kmem_cache_alloc [Tracepoint event] + kmem:mm_page_alloc [Tracepoint event] + kmem:mm_page_alloc_zone_locked [Tracepoint event] + kmem:mm_page_pcpu_drain [Tracepoint event] + kmem:mm_page_alloc_extfrag [Tracepoint event] + vmscan:mm_vmscan_kswapd_sleep [Tracepoint event] + vmscan:mm_vmscan_kswapd_wake [Tracepoint event] + vmscan:mm_vmscan_wakeup_kswapd [Tracepoint event] + vmscan:mm_vmscan_direct_reclaim_begin [Tracepoint event] + . + . + module:module_get [Tracepoint event] + module:module_put [Tracepoint event] + module:module_request [Tracepoint event] + sched:sched_kthread_stop [Tracepoint event] + sched:sched_wakeup [Tracepoint event] + sched:sched_wakeup_new [Tracepoint event] + sched:sched_process_fork [Tracepoint event] + sched:sched_process_exec [Tracepoint event] + sched:sched_stat_runtime [Tracepoint event] + rcu:rcu_utilization [Tracepoint event] + workqueue:workqueue_queue_work [Tracepoint event] + workqueue:workqueue_execute_end [Tracepoint event] + signal:signal_generate [Tracepoint event] + signal:signal_deliver [Tracepoint event] + timer:timer_init [Tracepoint event] + timer:timer_start [Tracepoint event] + timer:hrtimer_cancel [Tracepoint event] + timer:itimer_state [Tracepoint event] + timer:itimer_expire [Tracepoint event] + irq:irq_handler_entry [Tracepoint event] + irq:irq_handler_exit [Tracepoint event] + irq:softirq_entry [Tracepoint event] + irq:softirq_exit [Tracepoint event] + irq:softirq_raise [Tracepoint event] + printk:console [Tracepoint event] + task:task_newtask [Tracepoint event] + task:task_rename [Tracepoint event] + syscalls:sys_enter_socketcall [Tracepoint event] + syscalls:sys_exit_socketcall [Tracepoint event] + . + . + . + syscalls:sys_enter_unshare [Tracepoint event] + syscalls:sys_exit_unshare [Tracepoint event] + raw_syscalls:sys_enter [Tracepoint event] + raw_syscalls:sys_exit [Tracepoint event] + </literallayout> + </para> - <para> - The Yocto Project uses its own implementation of - <ulink url='http://www.bugzilla.org/about/'>Bugzilla</ulink> to track bugs. - Implementations of Bugzilla work well for group development because they track bugs and code - changes, can be used to communicate changes and problems with developers, can be used to - submit and review patches, and can be used to manage quality assurance. - The home page for the Yocto Project implementation of Bugzilla is - <ulink url='&YOCTO_BUGZILLA_URL;'>&YOCTO_BUGZILLA_URL;</ulink>. - </para> + <note> + Tying It Together: These are exactly the same set of events defined + by the trace event subsystem and exposed by + ftrace/tracecmd/kernelshark as files in + /sys/kernel/debug/tracing/events, by SystemTap as + kernel.trace("tracepoint_name") and (partially) accessed by LTTng. + </note> - <para> - Sometimes it is helpful to submit, investigate, or track a bug against the Yocto Project itself - such as when discovering an issue with some component of the build system that acts contrary - to the documentation or your expectations. - Following is the general procedure for submitting a new bug using the Yocto Project - Bugzilla. - You can find more information on defect management, bug tracking, and feature request - processes all accomplished through the Yocto Project Bugzilla on the wiki page - <ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>here</ulink>. - <orderedlist> - <listitem><para>Always use the Yocto Project implementation of Bugzilla to submit - a bug.</para></listitem> - <listitem><para>When submitting a new bug, be sure to choose the appropriate - Classification, Product, and Component for which the issue was found. - Defects for Yocto Project fall into one of six classifications: Yocto Project - Components, Infrastructure, Build System & Metadata, Documentation, - QA/Testing, and Runtime. - Each of these Classifications break down into multiple Products and, in some - cases, multiple Components.</para></listitem> - <listitem><para>Use the bug form to choose the correct Hardware and Architecture - for which the bug applies.</para></listitem> - <listitem><para>Indicate the Yocto Project version you were using when the issue - occurred.</para></listitem> - <listitem><para>Be sure to indicate the Severity of the bug. - Severity communicates how the bug impacted your work.</para></listitem> - <listitem><para>Provide a brief summary of the issue. - Try to limit your summary to just a line or two and be sure to capture the - essence of the issue.</para></listitem> - <listitem><para>Provide a detailed description of the issue. - You should provide as much detail as you can about the context, behavior, output, - and so forth that surround the issue. - You can even attach supporting files for output or log by using the "Add an attachment" - button.</para></listitem> - <listitem><para>Submit the bug by clicking the "Submit Bug" button.</para></listitem> - </orderedlist> - </para> -</section> + <para> + Only a subset of these would be of interest to us when looking at + this workload, so let's choose the most likely subsystems + (identified by the string before the colon in the Tracepoint events) + and do a 'perf stat' run using only those wildcarded subsystems: + <literallayout class='monospaced'> + root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + Performance counter stats for 'wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2': + + 23323 skb:kfree_skb + 0 skb:consume_skb + 49897 skb:skb_copy_datagram_iovec + 6217 net:net_dev_xmit + 6217 net:net_dev_queue + 7962 net:netif_receive_skb + 2 net:netif_rx + 8340 napi:napi_poll + 0 sched:sched_kthread_stop + 0 sched:sched_kthread_stop_ret + 3749 sched:sched_wakeup + 0 sched:sched_wakeup_new + 0 sched:sched_switch + 29 sched:sched_migrate_task + 0 sched:sched_process_free + 1 sched:sched_process_exit + 0 sched:sched_wait_task + 0 sched:sched_process_wait + 0 sched:sched_process_fork + 1 sched:sched_process_exec + 0 sched:sched_stat_wait + 2106519415641 sched:sched_stat_sleep + 0 sched:sched_stat_iowait + 147453613 sched:sched_stat_blocked + 12903026955 sched:sched_stat_runtime + 0 sched:sched_pi_setprio + 3574 workqueue:workqueue_queue_work + 3574 workqueue:workqueue_activate_work + 0 workqueue:workqueue_execute_start + 0 workqueue:workqueue_execute_end + 16631 irq:irq_handler_entry + 16631 irq:irq_handler_exit + 28521 irq:softirq_entry + 28521 irq:softirq_exit + 28728 irq:softirq_raise + 1 syscalls:sys_enter_sendmmsg + 1 syscalls:sys_exit_sendmmsg + 0 syscalls:sys_enter_recvmmsg + 0 syscalls:sys_exit_recvmmsg + 14 syscalls:sys_enter_socketcall + 14 syscalls:sys_exit_socketcall + . + . + . + 16965 syscalls:sys_enter_read + 16965 syscalls:sys_exit_read + 12854 syscalls:sys_enter_write + 12854 syscalls:sys_exit_write + . + . + . + + 58.029710972 seconds time elapsed + </literallayout> + Let's pick one of these tracepoints and tell perf to do a profile + using it as the sampling event: + <literallayout class='monospaced'> + root@crownbay:~# perf record -g -e sched:sched_wakeup wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + </literallayout> + </para> -<section id='how-to-submit-a-change'> - <title>How to Submit a Change</title> + <para> + <imagedata fileref="figures/sched-wakeup-profile.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> - <para> - Contributions to the Yocto Project and OpenEmbedded are very welcome. - Because the system is extremely configurable and flexible, we recognize that developers - will want to extend, configure or optimize it for their specific uses. - You should send patches to the appropriate mailing list so that they - can be reviewed and merged by the appropriate maintainer. - For a list of the Yocto Project and related mailing lists, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing lists</ulink>" section in - the Yocto Project Reference Manual. - </para> + <para> + The screenshot above shows the results of running a profile using + sched:sched_switch tracepoint, which shows the relative costs of + various paths to sched_wakeup (note that sched_wakeup is the + name of the tracepoint - it's actually defined just inside + ttwu_do_wakeup(), which accounts for the function name actually + displayed in the profile: + <literallayout class='monospaced'> + /* + * Mark the task runnable and perform wakeup-preemption. + */ + static void + ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags) + { + trace_sched_wakeup(p, true); + . + . + . + } + </literallayout> + A couple of the more interesting callchains are expanded and + displayed above, basically some network receive paths that + presumably end up waking up wget (busybox) when network data is + ready. + </para> - <para> - The following is some guidance on which mailing list to use for what type of change: - <itemizedlist> - <listitem><para>For changes to the core metadata, send your patch to the - <ulink url='&OE_LISTS_URL;/listinfo/openembedded-core'>openembedded-core</ulink> mailing list. - For example, a change to anything under the <filename>meta</filename> or - <filename>scripts</filename> directories - should be sent to this mailing list.</para></listitem> - <listitem><para>For changes to BitBake (anything under the <filename>bitbake</filename> - directory), send your patch to the - <ulink url='&OE_LISTS_URL;/listinfo/bitbake-devel'>bitbake-devel</ulink> mailing list.</para></listitem> - <listitem><para>For changes to <filename>meta-yocto</filename>, send your patch to the - <ulink url='&YOCTO_LISTS_URL;/listinfo/poky'>poky</ulink> mailing list.</para></listitem> - <listitem><para>For changes to other layers hosted on - <filename>yoctoproject.org</filename> (unless the - layer's documentation specifies otherwise), tools, and Yocto Project - documentation, use the - <ulink url='&YOCTO_LISTS_URL;/listinfo/yocto'>yocto</ulink> mailing list.</para></listitem> - <listitem><para>For additional recipes that do not fit into the core metadata, - you should determine which layer the recipe should go into and submit the - change in the manner recommended by the documentation (e.g. README) supplied - with the layer. If in doubt, please ask on the - <ulink url='&YOCTO_LISTS_URL;/listinfo/yocto'>yocto</ulink> or - <ulink url='&OE_LISTS_URL;/listinfo/openembedded-devel'>openembedded-devel</ulink> - mailing lists.</para></listitem> - </itemizedlist> - </para> + <para> + Note that because tracepoints are normally used for tracing, + the default sampling period for tracepoints is 1 i.e. for + tracepoints perf will sample on every event occurrence (this + can be changed using the -c option). This is in contrast to + hardware counters such as for example the default 'cycles' + hardware counter used for normal profiling, where sampling + periods are much higher (in the thousands) because profiling should + have as low an overhead as possible and sampling on every cycle w + ould be prohibitively expensive. + </para> + </section> - <para> - When you send a patch, be sure to include a "Signed-off-by:" - line in the same style as required by the Linux kernel. - Adding this line signifies that you, the submitter, have agreed to the Developer's Certificate of Origin 1.1 - as follows: - <literallayout class='monospaced'> - Developer's Certificate of Origin 1.1 - - By making a contribution to this project, I certify that: - - (a) The contribution was created in whole or in part by me and I - have the right to submit it under the open source license - indicated in the file; or - - (b) The contribution is based upon previous work that, to the best - of my knowledge, is covered under an appropriate open source - license and I have the right under that license to submit that - work with modifications, whether created in whole or in part - by me, under the same open source license (unless I am - permitted to submit under a different license), as indicated - in the file; or - - (c) The contribution was provided directly to me by some other - person who certified (a), (b) or (c) and I have not modified - it. - - (d) I understand and agree that this project and the contribution - are public and that a record of the contribution (including all - personal information I submit with it, including my sign-off) is - maintained indefinitely and may be redistributed consistent with - this project or the open source license(s) involved. - </literallayout> - </para> + <section id='using-perf-to-do-basic-tracing'> + <title>Using perf to do Basic Tracing</title> - <para> - In a collaborative environment, it is necessary to have some sort of standard - or method through which you submit changes. - Otherwise, things could get quite chaotic. - One general practice to follow is to make small, controlled changes. - Keeping changes small and isolated aids review, makes merging/rebasing easier - and keeps the change history clean when anyone needs to refer to it in future. - </para> + <para> + Profiling is a great tool for solving many problems or for + getting a high-level view of what's going on with a workload or + across the system. It is however by definition an approximation, + as suggested by the most prominent word associated with it, + 'sampling'. On the one hand, it allows a representative picture of + what's going on in the system to be cheaply taken, but on the other + hand, that cheapness limits its utility when that data suggests a + need to 'dive down' more deeply to discover what's really going + on. In such cases, the only way to see what's really going on is + to be able to look at (or summarize more intelligently) the + individual steps that go into the higher-level behavior exposed + by the coarse-grained profiling data. + </para> - <para> - When you make a commit, you must follow certain standards established by the - OpenEmbedded and Yocto Project development teams. - For each commit, you must provide a single-line summary of the change and you - should almost always provide a more detailed description of what you did (i.e. - the body of the commit message). - The only exceptions for not providing a detailed description would be if your - change is a simple, self-explanatory change that needs no further description - beyond the summary. - Here are the guidelines for composing a commit message: - <itemizedlist> - <listitem><para>Provide a single-line, short summary of the change. - This summary is typically viewable in the "shortlist" of changes. - Thus, providing something short and descriptive that gives the reader - a summary of the change is useful when viewing a list of many commits. - This should be prefixed by the recipe name (if changing a recipe), or - else the short form path to the file being changed. - </para></listitem> - <listitem><para>For the body of the commit message, provide detailed information - that describes what you changed, why you made the change, and the approach - you used. It may also be helpful if you mention how you tested the change. - Provide as much detail as you can in the body of the commit message. - </para></listitem> - <listitem><para>If the change addresses a specific bug or issue that is - associated with a bug-tracking ID, include a reference to that ID in - your detailed description. - For example, the Yocto Project uses a specific convention for bug - references - any commit that addresses a specific bug should include the - bug ID in the description (typically at the beginning) as follows: - <literallayout class='monospaced'> - [YOCTO #<bug-id>] - - <detailed description of change> - </literallayout></para></listitem> - Where <bug-id> is replaced with the specific bug ID from the - Yocto Project Bugzilla instance. - </itemizedlist> - </para> + <para> + As a concrete example, we can trace all the events we think might + be applicable to our workload: + <literallayout class='monospaced'> + root@crownbay:~# perf record -g -e skb:* -e net:* -e napi:* -e sched:sched_switch -e sched:sched_wakeup -e irq:* + -e syscalls:sys_enter_read -e syscalls:sys_exit_read -e syscalls:sys_enter_write -e syscalls:sys_exit_write + wget http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2 + </literallayout> + We can look at the raw trace output using 'perf script' with no + arguments: + <literallayout class='monospaced'> + root@crownbay:~# perf script + + perf 1262 [000] 11624.857082: sys_exit_read: 0x0 + perf 1262 [000] 11624.857193: sched_wakeup: comm=migration/0 pid=6 prio=0 success=1 target_cpu=000 + wget 1262 [001] 11624.858021: softirq_raise: vec=1 [action=TIMER] + wget 1262 [001] 11624.858074: softirq_entry: vec=1 [action=TIMER] + wget 1262 [001] 11624.858081: softirq_exit: vec=1 [action=TIMER] + wget 1262 [001] 11624.858166: sys_enter_read: fd: 0x0003, buf: 0xbf82c940, count: 0x0200 + wget 1262 [001] 11624.858177: sys_exit_read: 0x200 + wget 1262 [001] 11624.858878: kfree_skb: skbaddr=0xeb248d80 protocol=0 location=0xc15a5308 + wget 1262 [001] 11624.858945: kfree_skb: skbaddr=0xeb248000 protocol=0 location=0xc15a5308 + wget 1262 [001] 11624.859020: softirq_raise: vec=1 [action=TIMER] + wget 1262 [001] 11624.859076: softirq_entry: vec=1 [action=TIMER] + wget 1262 [001] 11624.859083: softirq_exit: vec=1 [action=TIMER] + wget 1262 [001] 11624.859167: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 + wget 1262 [001] 11624.859192: sys_exit_read: 0x1d7 + wget 1262 [001] 11624.859228: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 + wget 1262 [001] 11624.859233: sys_exit_read: 0x0 + wget 1262 [001] 11624.859573: sys_enter_read: fd: 0x0003, buf: 0xbf82c580, count: 0x0200 + wget 1262 [001] 11624.859584: sys_exit_read: 0x200 + wget 1262 [001] 11624.859864: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 + wget 1262 [001] 11624.859888: sys_exit_read: 0x400 + wget 1262 [001] 11624.859935: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 + wget 1262 [001] 11624.859944: sys_exit_read: 0x400 + </literallayout> + This gives us a detailed timestamped sequence of events that + occurred within the workload with respect to those events. + </para> - <para> - You can find more guidance on creating well-formed commit messages at this OpenEmbedded - wiki page: - <ulink url='&OE_HOME_URL;/wiki/Commit_Patch_Message_Guidelines'></ulink>. - </para> + <para> + In many ways, profiling can be viewed as a subset of tracing - + theoretically, if you have a set of trace events that's sufficient + to capture all the important aspects of a workload, you can derive + any of the results or views that a profiling run can. + </para> - <para> - Following are general instructions for both pushing changes upstream and for submitting - changes as patches. - </para> + <para> + Another aspect of traditional profiling is that while powerful in + many ways, it's limited by the granularity of the underlying data. + Profiling tools offer various ways of sorting and presenting the + sample data, which make it much more useful and amenable to user + experimentation, but in the end it can't be used in an open-ended + way to extract data that just isn't present as a consequence of + the fact that conceptually, most of it has been thrown away. + </para> - <section id='pushing-a-change-upstream'> - <title>Using Scripts to Push a Change Upstream and Request a Pull</title> + <para> + Full-blown detailed tracing data does however offer the opportunity + to manipulate and present the information collected during a + tracing run in an infinite variety of ways. + </para> <para> - The basic flow for pushing a change to an upstream "contrib" Git repository is as follows: - <itemizedlist> - <listitem><para>Make your changes in your local Git repository.</para></listitem> - <listitem><para>Stage your changes by using the <filename>git add</filename> - command on each file you changed.</para></listitem> - <listitem><para>Commit the change by using the <filename>git commit</filename> - command and push it to the "contrib" repository. - Be sure to provide a commit message that follows the project’s commit message standards - as described earlier.</para></listitem> - <listitem><para>Notify the maintainer that you have pushed a change by making a pull - request. - The Yocto Project provides two scripts that conveniently let you generate and send - pull requests to the Yocto Project. - These scripts are <filename>create-pull-request</filename> and - <filename>send-pull-request</filename>. - You can find these scripts in the <filename>scripts</filename> directory - within the <link linkend='source-directory'>Source Directory</link>.</para> - <para>Using these scripts correctly formats the requests without introducing any - whitespace or HTML formatting. - The maintainer that receives your patches needs to be able to save and apply them - directly from your emails. - Using these scripts is the preferred method for sending patches.</para> - <para>For help on using these scripts, simply provide the - <filename>-h</filename> argument as follows: - <literallayout class='monospaced'> - $ ~/poky/scripts/create-pull-request -h - $ ~/poky/scripts/send-pull-request -h - </literallayout></para></listitem> - </itemizedlist> + Another way to look at it is that there are only so many ways that + the 'primitive' counters can be used on their own to generate + interesting output; to get anything more complicated than simple + counts requires some amount of additional logic, which is typically + very specific to the problem at hand. For example, if we wanted to + make use of a 'counter' that maps to the value of the time + difference between when a process was scheduled to run on a + processor and the time it actually ran, we wouldn't expect such + a counter to exist on its own, but we could derive one called say + 'wakeup_latency' and use it to extract a useful view of that metric + from trace data. Likewise, we really can't figure out from standard + profiling tools how much data every process on the system reads and + writes, along with how many of those reads and writes fail + completely. If we have sufficient trace data, however, we could + with the right tools easily extract and present that information, + but we'd need something other than pre-canned profiling tools to + do that. </para> <para> - You can find general Git information on how to push a change upstream in the - <ulink url='http://book.git-scm.com/3_distributed_workflows.html'>Git Community Book</ulink>. + Luckily, there is general-purpose way to handle such needs, + called 'programming languages'. Making programming languages + easily available to apply to such problems given the specific + format of data is called a 'programming language binding' for + that data and language. Perf supports two programming language + bindings, one for Python and one for Perl. </para> - </section> - <section id='submitting-a-patch'> - <title>Using Email to Submit a Patch</title> + <note> + Tying It Together: Language bindings for manipulating and + aggregating trace data are of course not a new + idea. One of the first projects to do this was IBM's DProbes + dpcc compiler, an ANSI C compiler which targeted a low-level + assembly language running on an in-kernel interpreter on the + target system. This is exactly analagous to what Sun's DTrace + did, except that DTrace invented its own language for the purpose. + Systemtap, heavily inspired by DTrace, also created its own + one-off language, but rather than running the product on an + in-kernel interpreter, created an elaborate compiler-based + machinery to translate its language into kernel modules written + in C. + </note> + + <para> + Now that we have the trace data in perf.data, we can use + 'perf script -g' to generate a skeleton script with handlers + for the read/write entry/exit events we recorded: + <literallayout class='monospaced'> + root@crownbay:~# perf script -g python + generated Python script: perf-script.py + </literallayout> + The skeleton script simply creates a python function for each + event type in the perf.data file. The body of each function simply + prints the event name along with its parameters. For example: + <literallayout class='monospaced'> + def net__netif_rx(event_name, context, common_cpu, + common_secs, common_nsecs, common_pid, common_comm, + skbaddr, len, name): + print_header(event_name, common_cpu, common_secs, common_nsecs, + common_pid, common_comm) + + print "skbaddr=%u, len=%u, name=%s\n" % (skbaddr, len, name), + </literallayout> + We can run that script directly to print all of the events + contained in the perf.data file: + <literallayout class='monospaced'> + root@crownbay:~# perf script -s perf-script.py + + in trace_begin + syscalls__sys_exit_read 0 11624.857082795 1262 perf nr=3, ret=0 + sched__sched_wakeup 0 11624.857193498 1262 perf comm=migration/0, pid=6, prio=0, success=1, target_cpu=0 + irq__softirq_raise 1 11624.858021635 1262 wget vec=TIMER + irq__softirq_entry 1 11624.858074075 1262 wget vec=TIMER + irq__softirq_exit 1 11624.858081389 1262 wget vec=TIMER + syscalls__sys_enter_read 1 11624.858166434 1262 wget nr=3, fd=3, buf=3213019456, count=512 + syscalls__sys_exit_read 1 11624.858177924 1262 wget nr=3, ret=512 + skb__kfree_skb 1 11624.858878188 1262 wget skbaddr=3945041280, location=3243922184, protocol=0 + skb__kfree_skb 1 11624.858945608 1262 wget skbaddr=3945037824, location=3243922184, protocol=0 + irq__softirq_raise 1 11624.859020942 1262 wget vec=TIMER + irq__softirq_entry 1 11624.859076935 1262 wget vec=TIMER + irq__softirq_exit 1 11624.859083469 1262 wget vec=TIMER + syscalls__sys_enter_read 1 11624.859167565 1262 wget nr=3, fd=3, buf=3077701632, count=1024 + syscalls__sys_exit_read 1 11624.859192533 1262 wget nr=3, ret=471 + syscalls__sys_enter_read 1 11624.859228072 1262 wget nr=3, fd=3, buf=3077701632, count=1024 + syscalls__sys_exit_read 1 11624.859233707 1262 wget nr=3, ret=0 + syscalls__sys_enter_read 1 11624.859573008 1262 wget nr=3, fd=3, buf=3213018496, count=512 + syscalls__sys_exit_read 1 11624.859584818 1262 wget nr=3, ret=512 + syscalls__sys_enter_read 1 11624.859864562 1262 wget nr=3, fd=3, buf=3077701632, count=1024 + syscalls__sys_exit_read 1 11624.859888770 1262 wget nr=3, ret=1024 + syscalls__sys_enter_read 1 11624.859935140 1262 wget nr=3, fd=3, buf=3077701632, count=1024 + syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024 + </literallayout> + That in itself isn't very useful; after all, we can accomplish + pretty much the same thing by simply running 'perf script' + without arguments in the same directory as the perf.data file. + </para> <para> - You can submit patches without using the <filename>create-pull-request</filename> and - <filename>send-pull-request</filename> scripts described in the previous section. - Keep in mind, the preferred method is to use the scripts, however. + We can however replace the print statements in the generated + function bodies with whatever we want, and thereby make it + infinitely more useful. </para> <para> - Depending on the components changed, you need to submit the email to a specific - mailing list. - For some guidance on which mailing list to use, see the list in the - "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" section - earlier in this manual. - For a description of the available mailing lists, see - "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing Lists</ulink>" - section in the Yocto Project Reference Manual. + As a simple example, let's just replace the print statements in + the function bodies with a simple function that does nothing but + increment a per-event count. When the program is run against a + perf.data file, each time a particular event is encountered, + a tally is incremented for that event. For example: + <literallayout class='monospaced'> + def net__netif_rx(event_name, context, common_cpu, + common_secs, common_nsecs, common_pid, common_comm, + skbaddr, len, name): + inc_counts(event_name) + </literallayout> + Each event handler function in the generated code is modified + to do this. For convenience, we define a common function called + inc_counts() that each handler calls; inc_counts simply tallies + a count for each event using the 'counts' hash, which is a + specialized has function that does Perl-like autovivification, a + capability that's extremely useful for kinds of multi-level + aggregation commonly used in processing traces (see perf's + documentation on the Python language binding for details): + <literallayout class='monospaced'> + counts = autodict() + + def inc_counts(event_name): + try: + counts[event_name] += 1 + except TypeError: + counts[event_name] = 1 + </literallayout> + Finally, at the end of the trace processing run, we want to + print the result of all the per-event tallies. For that, we + use the special 'trace_end()' function: + <literallayout class='monospaced'> + def trace_end(): + for event_name, count in counts.iteritems(): + print "%-40s %10s\n" % (event_name, count) + </literallayout> + The end result is a summary of all the events recorded in the + trace: + <literallayout> + skb__skb_copy_datagram_iovec 13148 + irq__softirq_entry 4796 + irq__irq_handler_exit 3805 + irq__softirq_exit 4795 + syscalls__sys_enter_write 8990 + net__net_dev_xmit 652 + skb__kfree_skb 4047 + sched__sched_wakeup 1155 + irq__irq_handler_entry 3804 + irq__softirq_raise 4799 + net__net_dev_queue 652 + syscalls__sys_enter_read 17599 + net__netif_receive_skb 1743 + syscalls__sys_exit_read 17598 + net__netif_rx 2 + napi__napi_poll 1877 + syscalls__sys_exit_write 8990 + </literallayout> + Note that this is pretty much exactly the same information we get + from 'perf stat', which goes a little way to support the idea + mentioned previously that given the right kind of trace data, + higher-level profiling-type summaries can be derived from it. </para> <para> - Here is the general procedure on how to submit a patch through email without using the - scripts: - <itemizedlist> - <listitem><para>Make your changes in your local Git repository.</para></listitem> - <listitem><para>Stage your changes by using the <filename>git add</filename> - command on each file you changed.</para></listitem> - <listitem><para>Commit the change by using the - <filename>git commit --signoff</filename> command. - Using the <filename>--signoff</filename> option identifies you as the person - making the change and also satisfies the Developer's Certificate of - Origin (DCO) shown earlier.</para> - <para>When you form a commit you must follow certain standards established by the - Yocto Project development team. - See the earlier section - "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" - for Yocto Project commit message standards.</para></listitem> - <listitem><para>Format the commit into an email message. - To format commits, use the <filename>git format-patch</filename> command. - When you provide the command, you must include a revision list or a number of patches - as part of the command. - For example, these two commands each take the most recent single commit and - format it as an email message in the current directory: - <literallayout class='monospaced'> - $ git format-patch -1 - $ git format-patch HEAD~ - </literallayout></para> - <para>After the command is run, the current directory contains a - numbered <filename>.patch</filename> file for the commit.</para> - <para>If you provide several commits as part of the command, - the <filename>git format-patch</filename> command produces a numbered - series of files in the current directory – one for each commit. - If you have more than one patch, you should also use the - <filename>--cover</filename> option with the command, which generates a - cover letter as the first "patch" in the series. - You can then edit the cover letter to provide a description for - the series of patches. - For information on the <filename>git format-patch</filename> command, - see <filename>GIT_FORMAT_PATCH(1)</filename> displayed using the - <filename>man git-format-patch</filename> command.</para> - <note>If you are or will be a frequent contributor to the Yocto Project - or to OpenEmbedded, you might consider requesting a contrib area and the - necessary associated rights.</note></listitem> - <listitem><para>Import the files into your mail client by using the - <filename>git send-email</filename> command. - <note>In order to use <filename>git send-email</filename>, you must have the - the proper Git packages installed. - For Ubuntu and Fedora the package is <filename>git-email</filename>.</note></para> - <para>The <filename>git send-email</filename> command sends email by using a local - or remote Mail Transport Agent (MTA) such as - <filename>msmtp</filename>, <filename>sendmail</filename>, or through a direct - <filename>smtp</filename> configuration in your Git <filename>config</filename> - file. - If you are submitting patches through email only, it is very important - that you submit them without any whitespace or HTML formatting that - either you or your mailer introduces. - The maintainer that receives your patches needs to be able to save and - apply them directly from your emails. - A good way to verify that what you are sending will be applicable by the - maintainer is to do a dry run and send them to yourself and then - save and apply them as the maintainer would.</para> - <para>The <filename>git send-email</filename> command is the preferred method - for sending your patches since there is no risk of compromising whitespace - in the body of the message, which can occur when you use your own mail client. - The command also has several options that let you - specify recipients and perform further editing of the email message. - For information on how to use the <filename>git send-email</filename> command, - use the <filename>man git-send-email</filename> command.</para></listitem> - </itemizedlist> + Documentation on using the + <ulink url='http://linux.die.net/man/1/perf-script-python'>'perf script' python binding</ulink>. </para> </section> + + + + + </section> </chapter> <!-- |
