diff options
Diffstat (limited to 'README-NEW-AUTOBUILDER')
| -rw-r--r-- | README-NEW-AUTOBUILDER | 362 |
1 files changed, 0 insertions, 362 deletions
diff --git a/README-NEW-AUTOBUILDER b/README-NEW-AUTOBUILDER deleted file mode 100644 index c37c8c41..00000000 --- a/README-NEW-AUTOBUILDER +++ /dev/null @@ -1,362 +0,0 @@ -Introduction ------------- - -Prior versions of the yocto-autobuilder required an extensive knowledge of -buildbot as well as basic python. Over the years, as more targets were added -the main configuation file had become hard to maintain, harder to read and, -frankly, was not utilizing the power of buildbot. This document will explain -some of the rational behind the refactor, introduce end users to the new -configuration language, introduce the concept of customized buildsteps and -give some basic instruction on how to modify an autobuilder setup. - -TODO ------------- -- buildhistory is not functional -- still missing some needed build targets - - oe-core - - fsl - - p1022ds - - bunch of meta-intel - - -- minor UI cleanup for toggling of triggered layers -- bbpriority needs to work -- verify buildappsrcrev -- deploy stuff -- killing triggers. this has been a long standing issue. look into - -What is it ------------- - -Yocto Autobuilder is essentially some extension to the vanilla build bot. -This extension mainly addresses configuration file handling and Yocto specific -build steps. -For better maintainability, the Autobuilder (see Autobuilder.py located at -lib/python2.7/site-packages/autobuilder/), handles configuration from multiple -files. -Additional build steps such as CheckOutLayers.py or CreateBBLayersConf are -Yocto specific and simplify the bulders configuration. - -Setup ------------- - -Nothing here has changed much. - -git clone git://git.yoctoproject.org/yocto-autobuilder -cd yocto-autobuilder -. ./yocto-autobuilder-setup -yocto-start-autobuilder both - -Configuration -------------- - -In the bad old days, the yocto-autobuilder configuration was stored in a single -file. This file contained both build target definitions as well as helper code/ -custom buildsteps. This was obviously a "bad thing". We've now removed all -build target configuration out into the config/* directory - -Files ------ - -There are two needed files in this directory. One file, autobuilder.conf, is a -global definition file which sets various globals that controllers and workers need -to know about. This hasn't really changed from prior versions. - -One new file is yoctoAB.conf. This is the only required file needed for build -target definitions. However, as it would be a bad practice to have all build -targets within a single file, you may define build targets in other *.conf files. -As long as those conf files live within the config directory, the -yocto-autobuilder will automatically load them. This allows us to keep things -nice and neat. - -yoctoAB.conf: -------------- - -The main BuildSets section required for the configuration contains a single -property: - -[BuildSets] -order: ['nightly', 'nightly-x86'] - -The order property tells buildbot what order you wish to display build targets -in on the waterfall page. This can not be used with vanilla buildbot as there is -a small patch required to utilize this functionality. - -buildset configuration -------------- - -Each buildset must start with a section name followed by at three to four -properties: - -builders: A string or list of builders used to build the buildset - -repos: A list of dicts. Each item will contain a dict entry: - - 'name_of_repo': - {'repourl':'git://path.to/git_repo', - 'bbpriority':'1', - 'branch':'master'}} - - Either poky or openembedded must be the first repo listed, as that is - the base directory for building, and anything listed before it will be - deleted when the base directory is checked out. - -NOTE: bbpriority is currently not used - -props: A list of dicts. This is used when you need to add properties to a - builder. An example of this would be build-appliance which needs to have - a buildappsrcrev property passed so it knows which version from the poky - repo to pull. prop_type is basically buildbot scheduler properties with - the args sent as dict pairs. See: - http://buildbot.net/buildbot/docs/0.8.6/manual/cfg-schedulers.html#forcesched-parameters - -steps: A list of dicts. These steps can be either custom buildsteps or basic - buildbot buildsteps with args passed as dict pairs. Custom buildsteps - that we utilize are in lib/python2.7/site-packages/autobuilder/buildsteps. - - CreateBBLayersConf: This build step creates the bblayers.conf used by the - build system to determine which layers to use and where to find them. - - {'CreateBBLayersConf': {'buildprovider' : 'yocto', - 'bsplayer':'True', - 'bspprovider':'intel', - 'bbtextprepend':'text_to_prepend', - 'bbtextappend':'text_to_append', - 'layerdirs': ['layerdir1', 'layerdir2', ...]}} - - buildprovider: Optional and defaults to 'yocto'. If this is set to - 'yocto', then it adds the layers' 'meta', 'meta-yocto' and - 'meta-yocto-bsp' to BBLAYERS. If set to 'oe', then only 'meta' is - added to BBLAYERS. - - bsplayer: Optional and defaults to False. Set this to 'True' if you - want to use an established BSP layer and include its default layers. - - bspprovider: Required if bsplayer is set to True. Defines the BSP - provider so the default layers can be included. At this time only - 'intel', 'fsl-ppc' and 'fsl-arm' are recognized. - - bbtextprepend: Prepends the given text to bblayers.conf. Use \n in - the text to start a new line and always end the text with a \n. For - example: - 'bbtextprepend': '#Adding one comment line\n#Adding a second comment line\n' - - bbtextappend: Appends the given text to bblayers.conf. Same format - as bbtextprepend. - - layerdirs: Used to add layers to BBLAYERS. Layers are added in the - order specified and are added after the layers created by the - buildprovider. For example: - 'layerdirs': ['meta-openembedded/meta-oe', - 'meta-openembedded/meta-networking', - 'meta-openembedded/meta-gnome'] - will append the layers meta-openembedded/meta-oe, - meta-openembedded/meta-networking, and meta-openembedded/meta-gnome - to BBLAYERS. Note that this will override any default layers from a - bspprovider. - -scheduler: A list of dicts. Each item defines a scheduler associated with this - buildset: - - 'name_of_scheduler': - {'type': 'Nightly'} - - The scheduler type is specified by the 'type' property, where supported - values are 'Nightly' or 'SingleBranchScheduler', with 'Nightly' as the - default. Additional properties are used to configure the scheduler and - are type-specific: - - Nightly scheduler properties: - month: The month in which to start the build, with January = 1. - This defaults to '*', meaning every month. - dayOfWeek: The day of the week in which to start the build, with - Monday = 1. This defaults to '*', meaning every day. - hour: The hour of the day in which to start the build, with - 12:00 AM = 0. This must be set by the user. - minute: The minute of the hour in which to start the build, with - 0 to 59 as valid values. This must be set by the user. - - SingleBranchScheduler properties: - repository: the repository to attach the scheduler to; this - is the repo name from the 'repos' section; the branch which - the scheduler is attached to matches that in the repo - definition. - - stable-timer: how long (in seconds) to wait after change before - triggering build to allow for changes to settle. Defaults - to 60 seconds. - - change-user: the user name which the remote hook will use; the - default value is the repository name as per the 'repository' - property. - - change-password: password which the remote hook will use; the - default is the buildbot default 'changepw' -- if your - autobuilder is publically accessible, you want to keep this - secret to avoid injection of arbitrary changes into your - autobuilder. - - changesource: the type of changesource to use for the scheduler; - the default setting is a PBChangeSource. Only other valid - option is 'GitPoller' for a GitPoller changesource. See the - official Buildbot documentation at - http://docs.buildbot.net/latest/manual/cfg-changesources.html - for more information on changesources. - - interval: The interval between polling for changes. Only valid - for GitPoller changesources. Defaults to 300 seconds (five - minutes). - - Extra Notes on SingleBranchScheduler setup - ------------------------------------------ - - There is one SBS scheduler for each repository in the buildset, i.e., - if your buildset contains more repositories, and you want build - triggered by changes to any of them, you need to define a scheduler - for each. This also means that you can mix and match, e.g., only - do nightly rebuilds for some repos and immediate for others. - - The scheduler change filter is set up to watch changes only in the - single repository specified. If you are using a PBChangeSource, you - have to set up your repository hook to send the repository url in the - change set with the --repository argument to git_buildbot.py, e.g., - your git post-receive hook could look something like: - - #!/bin/sh - echo "$1 $2 $3" | \ - git_buildbot.py --repository="git://some_repo..." - - For local testing, you can use a post-merge hook instead, along - these lines: - - #!/bin/sh - PRE=$(git rev-parse 'HEAD@{1}') - POST=$(git rev-parse HEAD) - SYMNAME=$(git rev-parse --symbolic-full-name HEAD) - echo "$PRE $POST $SYMNAME" | \ - git_buildbot.py --repository="file:///where_your_repo_is" - - Because of the way builbot change sources work, your change user - names must not collide with the user names used by your workers. - - If you are using a GitPoller changesource and you specify a valid - repository but an non-existent branch, Buildbot will silently ignore - your scheduler. If a repository has changed recently but your build - has not triggered, validate that your repository and branch settings - are valid. - - For either a PBChangeSource or a GitPoller changesource, the - scheduler will not see the initial repository discovery as a change. - If you are using a SingleBranchScheduler schedule, it is good - practice to force a build to verify the buildset works correctly. - -Example of PBChangeSource: - -[nightly-x86] -builders: 'builder1' -repos: [{'poky': - {'repourl':'git://git.yoctoproject.org/poky', - 'bbpriority':'1', - 'branch':'master'}}, - {'meta-qt3': - {'repourl':'git://git.yoctoproject.org/meta-qt3', - 'bbpriority':'2', - 'branch':'master'}}] -steps: [{'SetDest':{}}, - {'CheckOutLayers': {}}, - {'RunPreamble': {}}, - {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'i686', - 'distro': 'poky'}}, - {'CreateBBLayersConf': {'buildprovider' : 'yocto'}}, - {'BuildImages': {'images': 'core-image-sato core-image-sato-dev'}}, - {'RunSanityTests': {'images': 'core-image-minimal core-image-sato'}}, - {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}] -scheduler: [{'dev-branch-scheduler' : - {'type':'SingleBranchScheduler', - 'repository':'poky', - 'stable-timer':30, - 'change-password':'secret_change_password'}}] - - -Example of MultiBranch GitPoller: - -[git-poller] -builders: 'example-worker' -repos: [{'poky-contrib': - {'repourl':'git://git.yoctoproject.org/poky-contrib', - 'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp'}, - 'branch':'tgraydon/poky'}}, - {'meta-qt3': - {'repourl':'git://git.yoctoproject.org/poky-contrib', - 'branch':'tgraydon/meta-qt3'}}] -steps: [{'SetDest':{}}, - {'CheckOutLayers': {}}, - {'RunPreamble': {}}, - {'GetDistroVersion' : {'distro': 'poky'}}, - {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'i686', - 'distro': 'poky'}}, - {'CreateBBLayersConf': {'buildprovider' : 'yocto'}}, - {'BuildImages': {'images': 'core-image-sato'}}, - {'PublishLayerTarballs':{}}, - {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}] -scheduler: [{'git-poller-scheduler': - {'type':'SingleBranchScheduler', - 'changesource':'GitPoller', - 'repository':'poky-contrib', - 'branch':'tgraydon/poky', - 'interval':3600}, - 'stable-timer':300}}] -scheduler: [{'git-poller-scheduler': - {'type':'SingleBranchScheduler', - 'changesource':'GitPoller', - 'repository':'poky-contrib', - 'branch':'tgraydon/meta-qt3', - 'interval':3600}, - 'stable-timer':300}}] - -Note that the 'change-password' property is not used for GitPoller. This -buildset config may also be found in the build-config.examples directory. - - -Adding Buildsteps -------------- -I've included the basic buildsteps required to do general building as well as an -example buildstep called HelloWorld.py. - -from buildbot.steps.shell import ShellCommand - -class HelloWorld(ShellCommand): - haltOnFailure = True - flunkOnFailure = True - name = "Hello World" - def __init__(self, factory, argdict=None, **kwargs): - self.firstname="" - self.lastname="" - self.factory = factory - for k, v in argdict.iteritems(): - if k=="name": - self.firstname=v - elif k=="lastname": - self.lastname=v - else: - setattr(self, k, v) - self.description = "Hello World" - self.command = "echo 'Hello World " + self.firstname + " " + self.lastname + "'" - ShellCommand.__init__(self, **kwargs) - - def describe(self, done=False): - description = ShellCommand.describe(self,done) - return description - -We see that this is an inherited class (from ShellCommand) that overrides the -__init__ and describe methods of ShellCommand. All custom buildsteps MUST override -__init__ of the inherited class, adding the argdict and kwargs properties to the -class. This allows us to pass arguments specific to the custom buildstep in via -an argument dictionary while allowing us to pass arguments in that will get passed -up to the inherited class as a keyworded, variable length argument list. - -We do this so that adding buildsteps is essentially drag and drop. By adding a -custom buildstep to lib/python2.7/site-packages/autobuilder/buildsteps you can -just reference it within your config file without making any changes to the core -Autobuilder codebase. - - |