| Age | Commit message (Collapse) | Author |
|---|
|
Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:
1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with
Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.
Item 2: While not supported it at least does not break the operation.
Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:
-P ./downloads works
-P ../downloads works
-P ../../downloads does not work
-P ./../downloads does not work
-P /home/user/downloads/../downloads does not work
In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')
Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".
This patch canonicalizes DL_DIR before it is passed to wget.
(Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9)
Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The save comes with an IO overhead which can slow down the rehash loop in bitbake
a lot. We only needed to do this when recipes were doing unihash cache copying. Now
they aren't doing that, drop this IO pain point.
(Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The code in OE-Core using this has been replaced with something more fit
for purpose. Drop these function calls as they were never a great idea in the
first place and cause IO slowdown for runqueue needing to sync the cache.
(Bitbake rev: 2c8fa57778c4bd2a5c48a60b701ac57de4289cb2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Upstream changed their urls, update our test to match.
(Bitbake rev: dc391b86540ec5e0a0f1d811c776a22d443b1c06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The default connection timeout for websockets is 10 seconds, so use the
provided client timeout instead (which defaults to 30 seconds).
(Bitbake rev: 23681775e5941e54ebead469addf708fca1e6beb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.
Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.
(Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we intercepted the file download to a temp file, we broke the
exist/size checks which need to happen before the rename. Correct
the ordering.
For some reason, python 3.12 exposes this problem in the selftests
differently to previous versions.
(Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When switching from normal mode to stream mode, skip calling
self._set_mode() again because this will cause a recursion into the
_set_mode() function and causes problems.
Also cleanup some of the error checking during this process
This bug affected when a client would attempt to switch from one stream
mode to another, and meant that the server would get an invalid message
from the client. This would cause the server to disconnect the client,
and the client would then reconnect in normal mode which was the mode it
wanted anyway and thus it would carry on without any errors. This made
the bug not visible on the client side, but resulting in a lot of
backtrace JSON decoding exceptions in the server logs.
(Bitbake rev: 1826bc41ab3369ac40034c5eaf698748b769b881)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Similar to looking up unihashes, use the batch API when checking if a
unihash exists to speed up lookups
(Bitbake rev: 0ac521ff37b578f7487bca0eccc7dc9e5974991b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Batching support on the client side has proven to be a much more
effective way of dealing with server latency than multiple client
connections and is also much nicer on the server, so drop the client
pool support from asyncrpc and the hash server
(Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Drops support for client pools, since batching support in the client
code has proven to be much more effective
(Bitbake rev: 85dafaf8e070459f7de7bfb37300d8b60a27002e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
cooker: We can call multiprocessing close() unconditionally and tweak a
comment give 3.8 is now the minimum version.
lib/bb: We can drop the logger addition code only needed before 3.6
asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the
conditional code.
(Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've seeing profiling tracebacks when parse errors occur during
profiling. Try and avoid these but not processing invalid profiles.
(Bitbake rev: 171bd9dd575307fbd61b5179ad86131d76add067)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 8e444cd9913d1ee0672b5583e263e5927c3221df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It makes sense to switch to a more recent version and keep up to date
with upstream changes and things like new python version support.
(Bitbake rev: f5462156036e71911c66d07dbf3303cde862785b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
SO_REUSEPORT is a socket option that allows multiple servers to listen
on the same TCP port, and the kernel will automatically load balance the
connections between them. This is particularly helpful for the hash
server since it runs in a single thread. To take advantage of a
multi-core server, multiple servers can be started in parallel with this
option (up to 1 per CPU) and the kernel will load balance between them.
(Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Uses the batching API of the client to reduce the effect of latency when
making multiple queries to the server
(Bitbake rev: a54734b4ac2ddb3bce004e576cf74d6ad6caf62a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changes the stream mode to do "batch" processing. This means that the
sending and reciving of messages is done simultaneously so that messages
can be sent as fast as possible without having to wait for each reply.
This allows multiple messages to be in flight at once, reducing the
effect of the round trip latency from the server.
(Bitbake rev: e768d0f17bdb97f6ff013ec3a41f182fecd47a55)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Task dependency data is becoming unwieldy with the number of indices it
contains. Convert it to use a named tuple instead, which allows members
to be indexed by a named property or an index (which allows it to retain
backward compatibility).
(Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Improve the rehash code to query unihashes in parallel since this is more
efficient on slower links.
(Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Improve the runqueue init code to call unihash queries in parallel since
this is faster and more efficient, particularly on slower links with longer
round trip times.
The call to the function from cooker is unneeded since that function calls
prepare() and hence this functionality will already have run, so drop
that obsolete call.
(Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The initial hash serve loop exits in the case where interrupts are present
but probably checks a bit too often. Tweak that and also allow the slow
rehash loop to break on interrupt, improving bitbake Ctrl+C response.
(Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With hashserve enabled, there are two slow paths/loops, one at initial runqueue
generation and also during the rehash process when new outhashes are found.
Add timing information at the hashserve log level for when these loops
take longer than 30s or 60s overall. This will leave evidence in the logs when
things are running particularly slowly.
(Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Based on the discution on https://lists.yoctoproject.org/g/toaster/message/6157
in some cases the value for Target_file could be missing and is needed to bypass
it to finish build.
(Bitbake rev: c60f6d20911632b41473f7c8577949be2f99ad80)
Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I don't know why there is hardcoded knowledge of S in the fetcher but there
shouldn't be and the OE unpack changes highlight this doing things it
shouldn't.
Drop the S reference and use rootdir which is the only place it should
be touching.
(Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Run them with "bitbake-selftest prserv.tests"
(Bitbake rev: 34287fbf3d6be813aa5b767f540e4662f0d8d18d)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Simplify the importone() hook:
- to make it independent from the "history" mode which is
client specific.
- remove the "history" parameter
- we want all values to be imported for binary
reproducibility purposes.
- using the store_value() function (which warrants
you don't save the same value twice and doesn't write
when you're using a read-only server) is enough.
(Bitbake rev: 000704a53470ab1ead840403b5531f22ebf1fd49)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a test_checksum_value() to test whether
a (version, pkgarch, checksum, value) entry already
exists in the database.
This is used to protect the store_value() function from
an error when trying to store a duplicate entry in the database.
Also check whether the current database is open in read-only mode.
(Bitbake rev: b7f6c085a7cf8ac83695242a0299e2d5f7abc69a)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove a possible race condition by allowing a read-only
server to create the PR table anyway. This avoids a failure
if both a read-only and read-write server try to access
an empty database at the same time.
(Bitbake rev: b171caec5ebbe579bf4b8b2005930240ae5c8ce2)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <jpewhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
sqlite3 can allow multiple processes to access the database
simultaneously, but it must be opened correctly. The key change is that
the database is no longer opened in "exclusive" mode (defaulting to
shared mode). In addition, the journal is set to "WAL" mode, as this is
the most efficient for dealing with simultaneous access between
different processes. In order to keep the database performance,
synchronous mode is set to "off". The WAL journal will protect against
incomplete transactions in any given client, however the database will
not be protected against unexpected power loss from the OS (which is a
fine trade off for performance, and also the same as the previous
implementation).
The use of a database cursor enabled to remove the _execute() wrapper.
The cursor automatically makes sure that the query happens in an atomic
transaction and commits when finished.
This also removes the need for a "dirty" flag for the database and
for explicit database syncing, which simplifies the code.
(Bitbake rev: 385833243c495dc68ec26a963136c1ced3f272d0)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Introduce a PRSERVER_UPSTREAM variable that makes the
local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an
upstream package (revision "x", in a way that gives the local
update priority (revision "x.y").
Update the calculation of the new revisions to support the
case when prior revisions are not integers, but have
an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py
for details about the calculation of the local revision.
This is done by going on supporting the "history" mode that
wasn't used so far (revisions can return to a previous historical value),
in addition to the default "no history" mode (revisions can never decrease).
Rather than storing the history mode in the database table
itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode
is now passed through the client requests. As a consequence, the
table name is now "PRMAIN", which is incompatible with what
was generated before, but avoids confusion if we kept the "PRMAIN_nohist"
name for both "history" and "no history" modes.
Update the server version to "2.0.0".
(Bitbake rev: 48857ec3e075791bd73d92747c609a0a4fda0e0c)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This script was the only user of this code.
(Bitbake rev: 19a5595e3f70d61fd6fa414f9fd5b413a02de37b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add missing declaration for the max_package_pr client hook
(Bitbake rev: 0d4443359ec38ff98b7fbae0b0948d14f74523ce)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tests quotes around `` and $() expansions, nested and multiple
expansions, and that escaped quotes are treated as characters by the
parser.
(Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 89712949de9476e4674864a8dcd6862fefe92eae)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current shell substitution mechanism only works without quotes. For
example:
var1=$(cmd1 ...)
Will work and add `cmd1` to the correspondind `run.do_*` file.
However, although quite common, this syntax is not supported:
var1="$(cmd1 ...)"
This commit adds this feature by adding a step to process_words() to
check whether we are dealing with quotes first, and by iterating on
what's between them to detect new shell substitution candidates. These
candidates are tested and parsed like before in the next step. The
original `part` being part of the candidates means the syntax
var1=$(cmd1 ...) is still valid.
(Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If an invalid provider is requested, error out early rather than trying
to build partial runqueue data structures as the taskdep UI will have
exited after seeing the bad provider.
(Bitbake rev: a478087998cb794cc4e31189b3ce07973d3949bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python 3.10 support is only available in websockets 10.0 and later:
https://github.com/python-websockets/websockets/commit/08d8011132ba038b3f6c4d591189b57af4c9f147
Update the version for this case. This avoids failures on Ubuntu 22.04.
(Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allow the use of new siggen API
(Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Checks that the minimum version of the websockets module is present, and
if not raises an ImportError. This allows the user to get earlier
feedback if using websockets is going to succeed
(Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server
(Bitbake rev: aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)
(Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.
[YOCTO #15473]
(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)
Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the cwd of the task is also a cleandirs, you would see warnings from bitbake
about being unable to obtain cwd during the task execution. Tweak the code
to detect this and avoid the warnings.
(Bitbake rev: 6c7fd60c10955b0f23f64b25b5b4e154eb22a8f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is actually rather easy: crate web API provides a json
with all the versions, for example:
https://crates.io/api/v1/crates/cargo-c/versions
(Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #15417]
When a layer adds a new dependency after it was added to a conf, it can not be
removed w/o this dependency in the setup. Even the dependent layer can not be
added, as the tinfoil setup will fail.
Adapt --force to not perform the tinfoil at all, the use will be at own risk,
i.e. the added layers might not parse properly afterwards.
This is not merged into the force option with -F as it even changes the loading of
plugins from other layers and is hence even more invasive as force. Instead
force can now be speciefied multiple times and is counted.
(Bitbake rev: 541fa7f582133949563e65f2d43c4b16e873e5c1)
Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.
(Bitbake rev: 5e84c13a6c594ed34c341849806657ddda206714)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the missing import. This bug was introduced with 1ab1d36c.
(Bitbake rev: 97ffe14311407f6e705ec24b70870ab32f0637b9)
Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 67a1aa8dbb3cb3a30fa7d697431ebb30323e4f28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Rudolf J Streif
Richard Purdie
Joshua Watt
Marlon Rodriguez Garcia
Michael Opdenacker
Antonin Godard
Kari Sivonen
Alexander Kanavin
Simone Weiß
Sven Schwermer