diff options
Diffstat (limited to 'documentation/ref-manual/ref-bitbake.xml')
| -rw-r--r-- | documentation/ref-manual/ref-bitbake.xml | 479 |
1 files changed, 0 insertions, 479 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml deleted file mode 100644 index 9e3e9cf35d2..00000000000 --- a/documentation/ref-manual/ref-bitbake.xml +++ /dev/null @@ -1,479 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<chapter id='ref-bitbake'> - - <title>BitBake</title> - - <para> - BitBake is a program written in Python that interprets the - <link linkend='metadata'>Metadata</link> used by - the OpenEmbedded build system. - At some point, developers wonder what actually happens when you enter: - <literallayout class='monospaced'> - $ bitbake core-image-sato - </literallayout> - </para> - - <para> - This chapter provides an overview of what happens behind the scenes from BitBake's perspective. - </para> - - <note> - BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships. - As such, it has no real knowledge of what the tasks being executed actually do. - BitBake just considers a list of tasks with dependencies and handles - <link linkend='metadata'>Metadata</link> - consisting of variables in a certain format that get passed to the tasks. - </note> - - <section id='ref-bitbake-parsing'> - <title>Parsing</title> - - <para> - BitBake parses configuration files, classes, and <filename>.bb</filename> files. - </para> - - <para> - The first thing BitBake does is look for the - <filename>bitbake.conf</filename> file. - This file resides in the - <link linkend='source-directory'>Source Directory</link> - within the <filename>meta/conf/</filename> directory. - BitBake finds it by examining its - <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment - variable and looking for the <filename>meta/conf/</filename> - directory. - </para> - - <para> - The <filename>bitbake.conf</filename> file lists other configuration - files to include from a <filename>conf/</filename> - directory below the directories listed in <filename>BBPATH</filename>. - In general, the most important configuration file from a user's perspective - is <filename>local.conf</filename>, which contains a user's customized - settings for the OpenEmbedded build environment. - Other notable configuration files are the distribution - configuration file (set by the - <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable) - and the machine configuration file - (set by the - <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable). - The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment - variables are both usually set in - the <filename>local.conf</filename> file. - Valid distribution - configuration files are available in the <filename>meta/conf/distro/</filename> directory - and valid machine configuration - files in the <filename>meta/conf/machine/</filename> directory. - Within the <filename>meta/conf/machine/include/</filename> - directory are various <filename>tune-*.inc</filename> configuration files that provide common - "tuning" settings specific to and shared between particular architectures and machines. - </para> - - <para> - After the parsing of the configuration files, some standard classes are included. - The <filename>base.bbclass</filename> file is always included. - Other classes that are specified in the configuration using the - <filename><link linkend='var-INHERIT'>INHERIT</link></filename> - variable are also included. - Class files are searched for in a <filename>classes</filename> subdirectory - under the paths in <filename>BBPATH</filename> in the same way as - configuration files. - </para> - - <para> - After classes are included, the variable - <filename><link linkend='var-BBFILES'>BBFILES</link></filename> - is set, usually in - <filename>local.conf</filename>, and defines the list of places to search for - <filename>.bb</filename> files. - By default, the <filename>BBFILES</filename> variable specifies the - <filename>meta/recipes-*/</filename> directory within Poky. - Adding extra content to <filename>BBFILES</filename> is best achieved through the use of - BitBake layers as described in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section of the Yocto Project Development Tasks Manual. - </para> - - <para> - BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and - stores the values of various variables. - In summary, for each <filename>.bb</filename> - file the configuration plus the base class of variables are set, followed - by the data in the <filename>.bb</filename> file - itself, followed by any inherit commands that - <filename>.bb</filename> file might contain. - </para> - - <para> - Because parsing <filename>.bb</filename> files is a time - consuming process, a cache is kept to speed up subsequent parsing. - This cache is invalid if the timestamp of the <filename>.bb</filename> - file itself changes, or if the timestamps of any of the include, - configuration files or class files on which the - <filename>.bb</filename> file depends change. - </para> - - <note> - <para> - You need to be aware of how BitBake parses curly braces. - If a recipe uses a closing curly brace within the function and - the character has no leading spaces, BitBake produces a parsing - error. - If you use a pair of curly brace in a shell function, the - closing curly brace must not be located at the start of the line - without leading spaces. - </para> - - <para> - Here is an example that causes BitBake to produce a parsing - error: - <literallayout class='monospaced'> - fakeroot create_shar() { - cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh - usage() - { - echo "test" - ###### The following "}" at the start of the line causes a parsing error ###### - } - EOF - } - </literallayout> - Writing the recipe this way avoids the error: - <literallayout class='monospaced'> - fakeroot create_shar() { - cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh - usage() - { - echo "test" - ######The following "}" with a leading space at the start of the line avoids the error ###### - } - EOF - } - </literallayout> - </para> - </note> - </section> - - <section id='ref-bitbake-providers'> - <title>Preferences and Providers</title> - - <para> - Once all the <filename>.bb</filename> files have been - parsed, BitBake starts to build the target (<filename>core-image-sato</filename> - in the previous section's example) and looks for providers of that target. - Once a provider is selected, BitBake resolves all the dependencies for - the target. - In the case of <filename>core-image-sato</filename>, it would lead to - <filename>packagegroup-core-x11-sato</filename>, - which in turn leads to recipes like <filename>matchbox-terminal</filename>, - <filename>pcmanfm</filename> and <filename>gthumb</filename>. - These recipes in turn depend on <filename>glibc</filename> and the toolchain. - </para> - - <para> - Sometimes a target might have multiple providers. - A common example is "virtual/kernel", which is provided by each kernel package. - Each machine often selects the best kernel provider by using a line similar to the - following in the machine configuration file: - </para> - - <literallayout class='monospaced'> - PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" - </literallayout> - - <para> - The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename> - is the provider with the same name as the target. - </para> - - <para> - Understanding how providers are chosen is made complicated by the fact - that multiple versions might exist. - BitBake defaults to the highest version of a provider. - Version comparisons are made using the same method as Debian. - You can use the - <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename> - variable to specify a particular version (usually in the distro configuration). - You can influence the order by using the - <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename> - variable. - By default, files have a preference of "0". - Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the - package unlikely to be used unless it is explicitly referenced. - Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used. - <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting. - <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package - versions until they have undergone sufficient testing to be considered stable. - </para> - - <para> - In summary, BitBake has created a list of providers, which is prioritized, for each target. - </para> - </section> - - <section id='ref-bitbake-dependencies'> - <title>Dependencies</title> - - <para> - Each target BitBake builds consists of multiple tasks such as - <filename>fetch</filename>, <filename>unpack</filename>, - <filename>patch</filename>, <filename>configure</filename>, - and <filename>compile</filename>. - For best performance on multi-core systems, BitBake considers each task as an independent - entity with its own set of dependencies. - </para> - - <para> - Dependencies are defined through several variables. - You can find information about variables BitBake uses in the - BitBake documentation, which is found in the - <filename>bitbake/doc/manual</filename> directory within the - <link linkend='source-directory'>Source Directory</link>. - At a basic level, it is sufficient to know that BitBake uses the - <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and - <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> - variables when calculating dependencies. - </para> - </section> - - <section id='ref-bitbake-tasklist'> - <title>The Task List</title> - - <para> - Based on the generated list of providers and the dependency information, - BitBake can now calculate exactly what tasks it needs to run and in what - order it needs to run them. - The build now starts with BitBake forking off threads up to the limit set in the - <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable. - BitBake continues to fork threads as long as there are tasks ready to run, - those tasks have all their dependencies met, and the thread threshold has not been - exceeded. - </para> - - <para> - It is worth noting that you can greatly speed up the build time by properly setting - the <filename>BB_NUMBER_THREADS</filename> variable. - See the - "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>" - section in the Yocto Project Quick Start for more information. - </para> - - <para> - As each task completes, a timestamp is written to the directory specified by the - <filename><link linkend='var-STAMP'>STAMP</link></filename> variable. - On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename> - directory and does not rerun - tasks that are already completed unless a timestamp is found to be invalid. - Currently, invalid timestamps are only considered on a per - <filename>.bb</filename> file basis. - So, for example, if the configure stamp has a timestamp greater than the - compile timestamp for a given target, then the compile task would rerun. - Running the compile task again, however, has no effect on other providers - that depend on that target. - This behavior could change or become configurable in future versions of BitBake. - </para> - - <note> - Some tasks are marked as "nostamp" tasks. - No timestamp file is created when these tasks are run. - Consequently, "nostamp" tasks are always rerun. - </note> - </section> - - <section id='ref-bitbake-runtask'> - <title>Running a Task</title> - - <para> - Tasks can either be a shell task or a Python task. - For shell tasks, BitBake writes a shell script to - <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script. - The generated shell script contains all the exported variables, and the shell functions - with all variables expanded. - Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>. - Looking at the expanded shell functions in the run file and the output in the log files - is a useful debugging technique. - </para> - - <para> - For Python tasks, BitBake executes the task internally and logs information to the - controlling terminal. - Future versions of BitBake will write the functions to files similar to the way - shell tasks are handled. - Logging will be handled in a way similar to shell tasks as well. - </para> - - <para> - Once all the tasks have been completed BitBake exits. - </para> - - <para> - When running a task, BitBake tightly controls the execution environment - of the build tasks to make sure unwanted contamination from the build machine - cannot influence the build. - Consequently, if you do want something to get passed into the build - task's environment, you must take a few steps: - <orderedlist> - <listitem><para>Tell BitBake to load what you want from the environment - into the data store. - You can do so through the <filename>BB_ENV_EXTRAWHITE</filename> - variable. - For example, assume you want to prevent the build system from - accessing your <filename>$HOME/.ccache</filename> directory. - The following command tells BitBake to load - <filename>CCACHE_DIR</filename> from the environment into the data - store: - <literallayout class='monospaced'> - export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR" - </literallayout></para></listitem> - <listitem><para>Tell BitBake to export what you have loaded into the - environment store to the task environment of every running task. - Loading something from the environment into the data store - (previous step) only makes it available in the datastore. - To export it to the task environment of every running task, - use a command similar to the following in your - <filename>local.conf</filename> or distro configuration file: - <literallayout class='monospaced'> - export CCACHE_DIR - </literallayout></para></listitem> - </orderedlist> - </para> - - <note> - A side effect of the previous steps is that BitBake records the variable - as a dependency of the build process in things like the shared state - checksums. - If doing so results in unnecessary rebuilds of tasks, you can whitelist the - variable so that the shared state code ignores the dependency when it creates - checksums. - For information on this process, see the - <filename>BB_HASHBASE_WHITELIST</filename> example in the - "<ulink url='&YOCTO_DOCS_CM_URL;#overview-checksums'>Checksums (Signatures)</ulink>" - section in the Yocto Project Concepts Manual. - </note> - </section> - - <section id='ref-bitbake-commandline'> - <title>BitBake Command Line</title> - - <para> - Following is the BitBake help output: - </para> - - <screen> -$ bitbake --help -Usage: bitbake [options] [recipename/target ...] - - Executes the specified task (default is 'build') for a given set of target recipes (.bb files). - It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which - will provide the layer, BBFILES and other configuration information. - -Options: - --version show program's version number and exit - -h, --help show this help message and exit - -b BUILDFILE, --buildfile=BUILDFILE - Execute tasks from a specific .bb recipe directly. - WARNING: Does not handle any dependencies from other - recipes. - -k, --continue Continue as much as possible after an error. While the - target that failed and anything depending on it cannot - be built, as much as possible will be built before - stopping. - -a, --tryaltconfigs Continue with builds by trying to use alternative - providers where possible. - -f, --force Force the specified targets/task to run (invalidating - any existing stamp file). - -c CMD, --cmd=CMD Specify the task to execute. The exact options - available depend on the metadata. Some examples might - be 'compile' or 'populate_sysroot' or 'listtasks' may - give a list of the tasks available. - -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP - Invalidate the stamp for the specified task such as - 'compile' and then run the default task for the - specified target(s). - -r PREFILE, --read=PREFILE - Read the specified file before bitbake.conf. - -R POSTFILE, --postread=POSTFILE - Read the specified file after bitbake.conf. - -v, --verbose Output more log message data to the terminal. - -D, --debug Increase the debug level. You can specify this more - than once. - -n, --dry-run Don't execute, just go through the motions. - -S, --dump-signatures - Don't execute, just dump out the signature - construction information. - -p, --parse-only Quit after parsing the BB recipes. - -s, --show-versions Show current and preferred versions of all recipes. - -e, --environment Show the global or per-package environment complete - with information about where variables were - set/changed. - -g, --graphviz Save dependency tree information for the specified - targets in the dot syntax. - -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED - Assume these dependencies don't exist and are already - provided (equivalent to ASSUME_PROVIDED). Useful to - make dependency graphs more appealing - -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS - Show debug logging for the specified logging domains - -P, --profile Profile the command and save reports. - -u UI, --ui=UI The user interface to use (e.g. knotty and taskexp). - -t SERVERTYPE, --servertype=SERVERTYPE - Choose which server to use, process or xmlrpc. - --revisions-changed Set the exit code depending on whether upstream - floating revisions have changed or not. - --server-only Run bitbake without a UI, only starting a server - (cooker) process. - -B BIND, --bind=BIND The name/address for the bitbake server to bind to. - --no-setscene Do not run any setscene tasks. sstate will be ignored - and everything needed, built. - --remote-server=REMOTE_SERVER - Connect to the specified server. - -m, --kill-server Terminate the remote server. - --observe-only Connect to a server as an observing-only client. - </screen> - </section> - - <section id='ref-bitbake-fetchers'> - <title>Fetchers</title> - - <para> - BitBake also contains a set of "fetcher" modules that allow - retrieval of source code from various types of sources. - For example, BitBake can get source code from a disk with the metadata, from websites, - from remote shell accounts, or from Source Code Management (SCM) systems - like <filename>cvs/subversion/git</filename>. - </para> - - <para> - Fetchers are usually triggered by entries in - <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>. - You can find information about the options and formats of entries for specific - fetchers in the BitBake manual located in the - <filename>bitbake/doc/manual</filename> directory of the - <link linkend='source-directory'>Source Directory</link>. - </para> - - <para> - One useful feature for certain Source Code Manager (SCM) fetchers - is the ability to "auto-update" when the upstream SCM changes - version. - Since this ability requires certain functionality from the SCM, - not all systems support it. - Currently Subversion, Bazaar and to a limited extent, Git support - the ability to "auto-update". - This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename> - variable. - See the - "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" - section in the Yocto Project Development Tasks Manual for more - information. - </para> - - </section> - -</chapter> -<!-- -vim: expandtab tw=80 ts=4 spell spelllang=en_gb ---> |