aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2018-10-17fetch2/git: provide information about missing sourcesUrs Fässler
Provide more information in the case the sources are not found in the unpack step. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: explicitly show the decision logic to select the source in codeUrs Fässler
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: prevent access to non-existing clonedirUrs Fässler
A user friendly error is throw when neither the clonedir nor fullshallow exist. Without the check, a difficult to interpret error is throw from within the fetch command. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2: extract the function which ensures that a valid symlink existsUrs Fässler
For better readability and future use, we extract the function which ensures that a given symlink exists. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: use intention revealing names for premirror testsUrs Fässler
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: add tests to capture existing behavior wrt. naming of git ↵Urs Fässler
shallow tarball The mapping of the URLs to the local shallow tarballs is not obvious. For easier understanding, we add this tests to explicitly showing the mapping. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: add tests to capture existing behavior wrt. naming of mirror tarballUrs Fässler
The mapping of the URLs to the local tarballs is not obvious. For easier understanding, we add this tests to explicitly showing the mapping. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2/git: add tests to capture existing behavior wrt. naming of clone ↵Urs Fässler
directories The mapping of the URLs to the local directory is not obvious. For easier understanding, we add this tests to explicitly showing the mapping. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17fetch2: avoid circular recursion with SRCPV in PRPeter Marko
Some recent changes broke SRCPV and workaround was introduced to avoid circular dependency if SRCPV is in PV. However there is still the same error if SRCPV is in PR. Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17taskexp: add focused search barTobias Olausson
Searching in the task explorer requires one to focus the task list. A readily visible and focused search bar makes searching intuitive. Signed-off-by: Tobias Olausson <tol@hms.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17layerindexlib: Fix various type checking errorsMark Hatle
In the list_obj function, we can't check if the requested object is 'in', the index data -- as it's actually an attribute of the object. Move to hasattr. The remaining items were incorrect usages of 'type' for class type comparison. Instead move to 'isinstance'. Remaing 'type' comparisons are still valid. The code was also reordered slightly to avoid a lot of: if not isinstance(x, y): ... else: ... reordering it removes the not and makes the code slightly easier to read. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17toaster: missing shutil importDavid Reyna
The "buildimport.py" script is missing the "import shutil" line, which causes a project import page failure. [YOCTO #12959] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17toaster: bump django version to match build toolsDavid Reyna
The version of django in the 2.6 release got bumped to 1.11.14. The toaster requirements file needs to be updated to accept it. [YOCTO #12958] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-17toaster: bad link for distro conf fileDavid Reyna
The distro source path link is missing "conf/distro/", and the display is using the machine link instead of the distro link. [YOCTO #12957] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-16data: Fix whitespace on _remove operationsRichard Purdie
We have some slightly odd behaviours with the current implementation of _remove operations. For example: TEST = " A B" TEST_remove = "C" would trigger TEST to become "A B" even thought it doesn't contain "C". In particular, this means that an inactive remove operator added in a bbappend could change the task checksum which is not desireable. Fix the operation to preserve whitespace, adding new tests to make this explict and test further corner cases. Also update the manual to match. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-16data_smart: Preserve history when renaming variablesRichard Purdie
Currently when a variable is renamed, all the variable history is lost. This makes it hard for users to understand/debug where values came from. Improve the code so that history is preserved across renamed variables. (Expanded variable key names are a special case of variable renaming) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-09main: Fix environment handling for UI module importsRichard Purdie
The environment was being cleared before the UI imports occurred which caused problems for graphical UIs like taskexp. The full environment was intended to be available to UI clients and it was only meant to be cleared for the server/cooker, so tweak the code order so this is the case. This fixes problems reported for taskexp. [YOCTO #12670] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04gitsm: Allow relative URIs when fetchingMark Hatle
Some repositories may specify a relative submodule path. If this happens, it is our responsibility to use the parents URL (ud) and handle any relative processing ourselves. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/gitsm: checkout submodules recursivelyLaurent Bonnans
The new fetcher did not run 'git submodule update' recursively. Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/gitsm: handle submodules nested inside subdirsLaurent Bonnans
The new gitsm fetcher assumed that submodules were living directly in the 'modules' directory, whereas they can be arbitrarily nested inside subdirectories. Solve it by first creating the parent of the destination directory for the symlink and copy steps. Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04toaster: custom image enable layer add, protect pre-cloned layersDavid Reyna
When creating custom image recipes, the layer add for new layers needs missing xhrLayerUrl data. Also, code is needed to check and inform user if the newly added layer has not been cloned yet, and provide helpful error message instead of the current frozen dialog. [YOCTO #12887] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04toaster: disable git remote check to allow for firewallsDavid Reyna
Toaster users behind firewalls that block "git" access usually require developers to add remap rules in their "~/.gitconfig" to remap GIT accesses to HTTP* access. However, there is a "git remote" test in Toaster that is not aware of such remaps, resulting in a false error. For now, disable this nice-to-have check to support this release, and re-enable when we can add remapping accommodations for this test. [YOCTO #12944] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04toaster: add 'thud' release to ToasterDavid Reyna
Toaster needs to include new YP-2.6 "Thud" release. [YOCTO #12943] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04toaster: error logs missing for cli buildsDavid Reyna
The method 'store_log_event' in 'buildinfohelper.py' always puts log messages from CLI builds into the backlog but never takes them out. The "close" method now forces all backlogged CLI events to be registered. [YOCTO #12813] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/gitsm: fix a typoLaurent Bonnans
'qbareclone' in place of 'bareclone' Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/git: use correct check to decide if the shallow tarball should be ↵Urs Fässler
unpacked The shallow_tarball check is never true due a check on the caller side. The tarball check is not related to the code on the caller side. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/git: prevent access to non-existing fullshallow tarballUrs Fässler
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04fetch2/git: improve readability of method need_updateUrs Fässler
To improve the readability we extract the different scenarios of why the clonedir needs an update. Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04bitbake-user-manual: Added section on multicong dependenciesScott Rifenbark
Created a new section to show how to handle dependencies when you are doing a multiple configuration build. Put it in the "Examples" section. Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
2018-10-04bitbake-user-manual: Added BBMULTICONFIG supportScott Rifenbark
The BBMULTICONFIG variable is a variable used for BitBake and was not documented in the BitBake Manual glossary. I added the definition. I also added the variable to the example in the section describing how to execute builds for multiple configurations. Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
2018-10-04bitbake-user-manual: Added new section on multiconfig buildsScott Rifenbark
Created a new example. Also added a figure for the separate configuration file hierarchy Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
2018-10-03test/fetch: use canonical GitLab URLsRoss Burton
GitLab returns a redirect from /foo to /foo.git which older releases of Git don't handle when cloning. These tests don't clone so the redirect works, but let's be consistant with the structure of these URLs and add the .git suffix. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25fetch2/gitsm.py: Rework the git submodule fetcherMark Hatle
The prior fetcher did not know how to work with MIRRORS, and did not honor BB_NO_NETWORK and similar. The new fetcher approach recursively calls 'gitsm' download on each submodule detected. This ensures that it will go throug the standard download process. Each downloaded submodule is then 'attached' to the original download in the 'modules' directory. This mimics the behavior of: git submodule init but there is no chance it will contact the network without permission. It then corrects upstream reference URIs. The unpack steps simply copies the items from the downloads to the destdir. Once copied the submodules are connected and we then run: git submodule update According to the git documentation, git submodule init can and will modify the project configuration and may connect to the network. Doing the work manually prevents this. (This manual process is allowed based on my reading of the documentation.) See: https://git-scm.com/book/en/v2/Git-Tools-Submodules The small change to the existing test is due to this new code always assuming the code is from a remote system, and not a 'local' repository. If this assumption proves to be incorrect -- code will need to be added to deal with local repositories without an upstream URI. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25lib/layerindexlib/tests/restapi.py: Fix test cases when BB_SKIP_NETTETS=yesMark Hatle
Change the way the network related tests are skipped, based on how the fetch tests are handled. We introduce a 'skipIfNoNetwork()' function that will report skipped tests through the standard unit test reporting system. [YOCTO #12928] Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25lib/bb/tests/parse.py: Test case was changing chdirMark Hatle
The test case was changing the current directory, but was never restoring it to the original location. This causes occasional failures in later test cases. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25lib/layerindexlib/tests/cooker.py: Fix topdir to use an absolute (real) pathMark Hatle
The test case needs to access test case files. Different versions of python may return absolute or relative locations in __file__. Use the same approach as other test cases in determining the location of the test files. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25utils: lockfile: Fix infinite loopIoan-Adrian Ratiu
A nasty corner case leads to a hang when utils.lockfile is called from oe-core's package-manager:deploy_dir_lock (in turn called from rootfs:_create further up the call stack) with "name" owned by root and the user running bitbake has no write access. Because this code runs under pseudo, the UID and EUID of the bitbake worker process are 0, so the os.access(dirname, os.W_OK) returns True i.e. it thinks the path is writable when in fact it's not writable. Only later when trying to open the file an Exception it thrown because the OS prohibits writing, but the Exception is ignored and the open is retried leading to an infinite loop. So this fix is to not ignore the "Permission Denied" exception. An alternative fix would be to replace the os.access() call with an try: open() except() at the beginning of the function. Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-21daemonize/build: Clean up /dev/null fd handlingRichard Purdie
At the end of bitbake selftest we see: sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r+' encoding='UTF-8'> Clean up the /dev/null handling to drop the unused entry in build.by and ensure the other open() calls are cleaned up. NULL was unused since http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/build.py?id=4a081b5a52e3d27da8d4b062f3fda292e8d8fb0a back in 2012. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-21COW: Fix StopIteration warningRichard Purdie
Fix the warning: WARNING: lib/bb/data_smart.py:235: DeprecationWarning: generator 'COWDictMeta.iter' raised StopIteration for k, v in self.variables.iteritems(): by using return from the generator, not raising StopIteration. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-07runqueue: Ensure setscene tasks that aren't covered get builtRichard Purdie
Running "bitbake gconf-native -c cleansstate; bitbake core-image-sato:do_populate_sdk" results in a build where it fails to find gconf-native and fails to build it, merrily trying to build the SDK without gconf being present. The issue is the missing setscene tasks are effectively ignored as the later code in runqueue thinks that since other sstate tasks are present, these 'cover' the missing one. In reality we need to call BB_SETSCENE_DEPVALID to make that decision. To do that we need a "reduced" setscene dependency graph which we don't have in main task graph context. Since that was already done in setscene, we should just assume anything in the non-covered list needs to be built. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-07cooker/runqueue: Turn universe warnings into verbnotesRichard Purdie
If the user puts universe on the commandline, they don't really want warnings so use the new verbnote level instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-07msg: Add explicit verbnote log levelRichard Purdie
It has become apparant we need a log level which reaches the console but isn't a warning/error. Add "verbnote" as a way of doing this, behaves as a note but with a higher priority. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-04server/process: Various server startup logging fixesRichard Purdie
There were various problems in the server startup loggin: a) stdout/stderr were not being flushed before forking which could potentially duplicate output b) there were separate buffers for stdout/stderr leading to confusing logs where the entries could be reordered. This was particularly confusing due to the separator the logs use to idendify new messages c) an fd wasn't being closed during server startup meaning if the server failed to start, the closed fd wasn't detected as it was held open by the other reference d) If the pipe was detected as being closed, the code incorrectly retried server startup e) The event code would remap stdout/stderr without flushing them, leading to lose log messages Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-04cooker.py: Fix incorrect bb files matched warningMark Hatle
In the case of a sublayer of an existing layer, where the sublayer and main layer share a path, the system may not match the paths properly resulting in: No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer' because it has already matched the main layer. Fix this issue by sorting the collection items based on the pattern, using longest to shortest. Obviously regex wildcards could still be an issue but these are typically not used, so this simply fix should work in the existing cases. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28tests/fetch: Update gnome.org urls after upstream changesRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28fetcher: Fixed remote removal not throwing exception.Paulo Neves
Before this fix it is assumed that the removal of the remote can only fail because there is not remote to remove. This is a false assumption. Example error which would be ignored: git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output: Note: A branch outside the refs/remotes/ hierarchy was not removed; to delete it, use: git branch -d master error: could not lock config file config error: Could not remove config section 'remote.origin' Due to the masking of this error a stranger error will be presented to the user, because this time we do not mask the exception: git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output: fatal: remote origin already exists. The most likely reason that the remote cannot be removed nor modified is that the DL_DIR/git2 does not have permissions compatible with the user running bitbake. This commit fixes: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728 Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28toaster: enable project import and merged Toaster settingsDavid Reyna
Enable the ability to select an existing build directory into Toaster. This opens to the user the backend features of 12823, for command line user compatibility. Enable the ability to select saving Toaster settings in the regular "bblayers.conf" and "local.conf" instead of the default "toaster_bblayers.conf" and "toaster.conf". This opens to the user the backend features of 12821, for command line user compatibility. [YOCTO #12902] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28toaster/layerdetails.js: don't hide local layer infoAwais Belal
The local layer info (provided through custom fixtures) should not be hidden. It is better to handle it in the same manner as an imported layer, otherwise the layer path and dependency info is not shown. The layer editing fields are handled in the html side of things appropriately so this does not harm that implementation. [YOCTO #12891] Signed-off-by: Awais Belal <awais_belal@mentor.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28toaster/models.py: allow local paths for custom recipe's baseAwais Belal
In a case where the layer source is local only and the recipe is not yet built, we can search for the path with layer's local_source_dir, and if available that should be used rather than just skipping the scenario. [YOCTO #12891] Signed-off-by: Awais Belal <awais_belal@mentor.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28toaster/checksettings: allow CUSTOM_XML_ONLY setting through envAwais Belal
This change allows the CUSTOM_XML_ONLY toaster setting to be provided through the environment so the user can do this without mingling with the settings.xml, for scenarios where modifying settings.xml is not achievable. [YOCTO #12891] Signed-off-by: Awais Belal <awais_belal@mentor.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>