| Age | Commit message (Collapse) | Author |
|---|
|
sstate mirrors can be very helpful, but in normal operation clients will
make a bunch of queries to the server for hash files that don't exist.
This can put a lot of undue stress on the server, since negative lookups
can often be expensive (especially when using a CDN).
To help mitigate this, the hash equivalence server can be used to cull
the lookups that are unlikely to succeed. This is done by first querying
if the server is aware of a given unihash (which the server is optimized
to do quickly), and only if the server thinks a given unihash exists will
an attempt be made to pull it from an sstate mirror.
Note that this _only_ affect sstate mirrors; local sstate caches are
assumed to be fast for negative lookups, but we may be able to extend
this to cover the local cache also in the future if desired.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Adds API to query if unihashes are known to the server in parallel
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Implements the new API required for querying unihashes in parallel
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.
The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.
NOTE: This requires changes to downstream siggen classes to implement
the new API. Specifically, they should rename any implementation of
get_unihash() -> get_cached_unihash()
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Adds two new subcommands to maintain the hash equivalence server with an
sstate cache.
The first subcommand is `prune-sstate` which will remove any sstate
files where the unihash is unknown to the hash equivalence server (or
any upstream).
The second subcommand `gc` does a garbage collection on the hash
equivalence server, removing any hash entries that don't have a
corresponding sstate file
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Implements a Client Pool derived from the AsyncRPC client pool that
allows querying for multiple equivalent hashes in parallel
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Adds API to check if the server is aware of the existence of a given
unihash. This can be used as an optimization for sstate where a client
can query the hash equivalence server to check if a unihash exists
before querying the sstate cache. If the hash server isn't aware of the
existence of a unihash, then there is very likely not a matching sstate
object, so this should be able to significantly cut down on the number
of negative hits on the sstate cache.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Use the _execute() helper to execute queries. This helper does the
logging of the statement that was being done manually everywhere.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
Adds support for removing unused unihashes from the database. This is
done using a "mark and sweep" style of garbage collection where a
collection is started by marking which unihashes should be kept in the
database, then performing a sweep to remove any unmarked hashes.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
|
|
This patch adds the following variables to the allow-list, which are
used in the "IAM roles for AWS when using the GitLab chart":
- AWS_ROLE_ARN
- AWS_WEB_IDENTITY_TOKEN_FILE
These variables are set in the CI job environment and are needed to
access the sstate cache artifacts in a connected S3 bucket.
[1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html
Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com>
(Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
docstrings
Docstring for git fetcher is missing destsuffix and subpath parameters,
so add them
(Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When downloading a Git repository containing an LFS, the Git hooks
are not always initialized correctly to perform the download. This
change updates the Git downloader to run the "git lfs install" command
in order to "smudge" the LFS files. The tests have also been updated
to do checks for situations in which git lfs is not installed, as
the application was required to be installed for any of the tests
to run previously.
The Git LFS functionality was working to some degree previously,
but this change also updates the fetcher to allow LFS downloads for
nobranch SRC_URIs.
(Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217)
Signed-off-by: Desone Burns II <dburns@seegrid.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Modify the django settings to make cache, logs and databases save
to a configurable directory with TOASTER_DJANGO_TMPDIR instead of
the machine root level /tmp directory.
In addition, add this variable to the passthroughs for tox.
This change is made to facilitate cleaning test files for runs on
the autobuilder and avoid errors involving the persisted database.
(Bitbake rev: 887aaa91fd54ef24fc3cce42d584b381e37d3343)
Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
variables
Toaster build tests create new build environments which do not utlize
any existing build environment variables.
In particular, DL_DIR and SSTATE_DIR from an existing environment can
be passed in to allow for faster builds.
Adding these as passthroughs specifically resolves slow builds
related to the autobuilder integrations of the toaster test suite.
(Bitbake rev: 58cc3a4671878a425f52f9ec21636e366356bfc2)
Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 3f335913bbbabf48db1749d197c3bfaac9fb7236.
This CVE shouldn't affect master, it is for binutils versions 2.34
thru 2.38, while master is 2.41
See: https://nvd.nist.gov/vuln/detail/CVE-2022-47007
Later in commit:
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=19cacf672930cee20feaf1f3468e3d5ac3099ffd
elements of the CVE fix are reverted deliberately so match upstream for this
and drop this 'fix'.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
LLVM doesn't actually need a native Python3 as the host Python is
sufficient, but as it then looks at the host for optional Python
dependencies explicitly disable their detection so that rust-llvm
remains deterministic. As this is a minimal LLVM for Rust, we don't
need the optviewer tool.
(From OE-Core rev: c6971d9d9a9d9a83615508d244778ddc89a94b8a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 3f335913bbbabf48db1749d197c3bfaac9fb7236)
Signed-off-by: Deepthi Hemraj <Deepthi.Hemraj@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Allow glslang to find external spirv-tools to build.
* Vulkan-validation-layers has been split. So a new recipe
Vulkan-Utility-Libraries has been added and vulkan-validation-layers now
depends on it.
(From OE-Core rev: 63bdeb74b133840f9d6f55093bf16f5a8a5d0ab7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Running `devtool search gcc` would result in errors for virtual recipes.
The reason was the newly introduced :layer-<layername> override.
When tinfoil parsed the recipes using the parseRecipeFile() function, it
failed to properly identify the layername for virtual recipes, which
resulted in the errors.
Add tests that exercise these code paths through parseRecipeFile().
(From OE-Core rev: cf7f34f93f9f90279412ea57d6d5930854d411ba)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 9d0aab7e94a5862e2a7003ebbc5704ec713dfb28)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 2d1364a9a590cfb84bb47ca38a6b626769e960a8)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: b7f8e818a83772ecab14655f9e0fba56960ec282)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: bfddff1bd6503729eceb905412f9c5c944a59d55)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 4176272fe33039302f0685ba96f5202a113bce7f)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 5b5c9d86e7face83b7a5c21ca88056daaaede5e4)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 381d785bd35c11ea766c771663a1272fc359bb2c)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: f6be55ef1e862d93d8275fb5792e6c66b0a0beee)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The Source: variable is generated from FILE but this is excluded from checksums
normally which results in a reproduciubility issue when the filename changes.
Add in a dependency by reworking the code a little to avoid this.
(From OE-Core rev: 3ea7da76c6930031a0071069027b1d71f737fbc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As this recipe tracks the release tags we can embed the PV in the
filename.
(From OE-Core rev: 6eae2b57d01be9906d0933be22a99e370b529b2e)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By setting this, event data from during the build is saved out. This can
be used for analytic including toaster to allow builds to be examined
later. This will use some disk space but it is worth it for the ability
to look into what happened during the build. Users can set the value to
something else in local.conf to override if they wish.
(From OE-Core rev: 4719b68bb9e3cccb85aed55350e53a439360abcb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I believe other fixes and cleanup means the CDN tests are now working
correctly, so enable it.
(From OE-Core rev: baabe012e158e387a692b234920eeab977c27440)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
image recipe
Test the show rows feature in image recipe table on project page
(Bitbake rev: 0afbce01f8b037adb9311f63dcb63a2d180a5373)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
recipe
BUILD IMAGE RECIPES: toastergui/project/{id}/images/
Test the edit column feature in image recipe table on project page
(Bitbake rev: 005f8e4a875d280268e0369d26aaaa9e5a65fd68)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BUILD IMAGE RECIPES: toastergui/project/{id}/images/
Test project page section images
- Check image recipes are displayed
- Check search input
- Check image recipe build button works
- Check image recipe table features(show/hide column, pagination)
(Bitbake rev: 9aa0aad6812730d4ffaa9b3adfae041176274475)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
image"
Test project page tab "New custom image" when no custom image
(Bitbake rev: fd8fe1d3b25679032a43c1dd25a945318e573cef)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 857e6fdefa655fa73460c9bb008dc08abb4c3107)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Check if project config tab right section contains five blocks:
- Machine:
- check 'Machine' is displayed
- check can change Machine
- Distro:
- check 'Distro' is displayed
- check can change Distro
- Most built recipes:
- check 'Most built recipes' is displayed
- check can select a recipe and build it
- Project release:
- check 'Project release' is displayed
- check project has right release displayed
- Layers:
- check can add a layer if exists
- check at least three layers are displayed
- openembedded-core
- meta-poky
- meta-yocto-bsp
(Bitbake rev: cbf0df0604e6cb37aa8c4433e9af3c321be80010)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
- Check if the menu is displayed and contains the right elements:
- Configuration
- COMPATIBLE METADATA
- Custom images
- Image recipes
- Software recipes
- Machines
- Layers
- Distro
- EXTRA CONFIGURATION
- Bitbake variables
- Actions
- Delete project
(Bitbake rev: a12fe481aac82e1feebdb56afb67ba77a8904995)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
displayed
Test project tabs:
- "configuration" tab
- "Builds" tab
- "Import layers" tab
- "New custom image" tab
Check search box used to build recipes
(Bitbake rev: 13a55ebe630ad20e8ab4cdcb3dc6dcbf4fa7243a)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Test edit project name:
- Click on "Edit" icon button
- Change project name
- Click on "Save" button
- Check project name is changed
(Bitbake rev: 0bd0bbf8df2a1043d5c768a70e4c4fabd5195227)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
right items
Check page header in project page:
- AT LEFT -> Logo of Yocto project, displayed, clickable
- "Toaster"+" Information icon", displayed, clickable
- "Server Icon" + "All builds", displayed, clickable
- "Directory Icon" + "All projects", displayed, clickable
- "Book Icon" + "Documentation", displayed, clickable
- AT RIGHT -> button "New project", displayed, clickable
(Bitbake rev: 5b2b8fe16e2adb45ca88165162c65c0f7f9a0755)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed
- Select import,
- Remove trailing whitespace,
- Set excepted 2lines after imports
(Bitbake rev: f0c91629fad2e556cd429f561cc244aef4b7b066)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Test the show rows feature in the projects table on the all projects page
(Bitbake rev: d899a3cf9e186f78780415195c734fdef359d08d)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Test the edit column feature in the projects table on the all projects page
(Bitbake rev: 16e05122b7298c449bf6cec1bcae75c3fb5d87db)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Test the search box in the all project table on the all projects page
(Bitbake rev: a2c0e65e90619268da916a235a97df890697d22f)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Also correct the description of variant2virtual().
(Bitbake rev: d766e9bd22ec6392fbf1694eea5032b9d09f1949)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In _parse_recipe(), bb_data was initially a datastore object, but then
after the call to bb.parse.handle() it was all of a sudden turned into a
dict of variant:datastore pairs. At the same time, parseRecipeVariants()
used bb_data for datastore objects, while parseRecipe() used it for the
dicts of variant:datastore pairs.
Change these functions to consistently use "bb_data" for datastore
objects and use "datastores" for the dicts of variant:datastore pairs.
(Bitbake rev: ed0f94c037476efd465db87726b2480406076beb)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Running `devtool search gcc` would result in errors like this for
virtual recipes:
ERROR: When reparsing
.../meta/recipes-devtools/gcc/libgcc-initial_13.2.bb:do_populate_sysroot,
the basehash value changed from
b1cd809ed98cef9db0fb1b17d34c4083e739c336f9d5619b89715b0294d81af5 to
44c2f92781dc4a20e98b7bb4724e204e64b101905fa75e71241a574b725997dc. The
metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Snone
ERROR: Then:
ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Sprintdiff
The reason was the newly introduced :layer-<layername> override, which
is used, e.g., in meta/classes-global/insane.bbclass to add the
patch-status QA test only for the meta layer:
ERROR_QA:append:layer-core = " patch-status"
When tinfoil parsed the recipes using the parseRecipeFile() function, it
failed to properly identify the layername for virtual recipes, which
resulted in the error above.
The correct thing to do is to make parseRecipeFile() call
bb.cache.virtualfn2realfn() to convert the virtual filename into a real
filename and virtual class.
(Bitbake rev: da2aed134412f5de04d7b540f92d735983ad0108)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we expand the variable AUTOREV in OE-Core, it triggers side effects in the
fetcher. The situation isn't ideal and needs improvement but this breaks
and is blocking enabling BB_DEFAULT_EVENTLOG.
Hack around the issue for now so we unblock things until we can work out
a better plan for how to improve AUTOREV support.
(Bitbake rev: cb9b6530f3d12c56a8b48847af2e7461924205d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Joshua Watt
Felix Moessbauer
Julien Stephan
Desone Burns
Alexander Lussier-Cullen
Richard Purdie
Ross Burton
Deepthi Hemraj
Anuj Mittal
Peter Kjellerstedt
Alassane Yattara