Age | Commit message (Collapse) | Author |
|
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).
Loosely based on an earlier prototype implementation by Qing He.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you send back a string here you get "TypeError: 'str' does not
support the buffer interface" errors in bitbake-cookerdaemon.log and
"IncompleteRead(0 bytes read, 22 more expected)" errors on the client
side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we're in observe-only mode then we cannot run commands that would
affect the server's state, including getSetVariable, so prevent that
from being called in observe-only mode.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:
Level Option Result
----- ------ ----------------------------------------
0 Print usual output
1 -q Only show progress and warnings or above
2 -qq Only show warnings or above
3+ -qqq Only show errors
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no "datasmart" member, only dataroot. This dates back to the
original implementation of variable history support - it's surprising we
haven't noticed the issue until now, but I guess it's rare to change a
copy of a datastore in a manner which using the old reference would
cause an issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove the template context to JSON decorator function as this is
deprecated by having a proper REST API.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update js tests so that we use the new Layer REST API.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Switch the front end to use the proper REST API for retrieving layer
information.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the url to the Layer typeahead so that this can be used later on by
the front end code to look up layer details.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a get API for returning information about layers.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the front end mechanism to load the typeahead for the git revision
field on importing a layer.
Also fix one indentation issue and update the js test.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we're importing a layer it's useful to suggest available git revisions of the
layers. This uses git ls-remote to fetch the revisions and then filter on this.
Caching is added for this typeahead to avoid having to fetch this
information multiple times in a single session.
[YOCTO #8429]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix flake8 issues and remove redundant __init__ function definitions
from typeaheads (likely a copy and paste error).
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Switch the importlayer.js to use the new REST API for importing a new layer.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add layer adding REST api and remove old views method.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow passing none as the project id, this is convenient for layers
which belong to projects already and therefore have their own project
field.
Add documentation string to function
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a visual indicator when the data is changing in the table, for instance if
it's being re-ordered, searched or paginated.
[YOCTO #10104]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #10444]
Added a new section about syntax called "Line Joining". The
section describes how you can use the backslash character (\)
to joing stuff.
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
|
|
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
|
|
Now that we use python3, we should refer to pip3, not pip.
[YOCTO #10774]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If many files change and the inotify queue overflows, rather than print
a traceback, invalidate the caches and warn the user.
[YOCTO #10676]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The processing of the "do_" prefix to tasks is currently inconsistent
and has resulted in "bitbake world -g" being broken as task prefixes
don't get handled correctly.
Make the "do_" task prefix handling consistent through various codepaths.
[YOCTO #10651]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid errors like:
ERROR: Exception handler error: 'NoneType' object has no attribute 'decode'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore
generate and send this data. In this case its a "pre collapsed" tree
but that is fine for the use cases in question.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to know the task hash in code using TASKDEPDATA so add this
data to the data structure. The recipe specific sysroots in OE
need this data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Looking further at the CPU loads on systems running large numbers of tasks,
the following things helps performance:
* Loop on waitpid until there are no processes still waiting
* Using select to wait for the cooker pipe to be writable before writing
avoiding pointless 100% cpu usage
* Only reading from worker pipes that select highlights are readable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules
have moved from _base to base. Handle this, while staying compatible
with earlier versions.
Signed-off-by: Daniel Lublin <daniel@lublin.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.
[YOCTO #10740]
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds the ability to specify a Selenium Docker container server as
a driver. This allows for repeatable tests independent of host.
Currently we assume you are using the Firefox container. Instructions
are located in the README in tests/browser.
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The parameters to Python string formatting need to be inside a tuple.
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.
This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
monitoring (probably not that important)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There's no need to encode and decode the hash key as a single string,
a tuple works just fine. Iterating over entries can be written more
concisely.
Entries in the stat results are integers, not floating point values.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:
- When there is just one long running task which fills up the disk,
the previous approach did not notice that until after the completion
of the task because _execute_runqueue() only gets called on task
state changes. As a result, aborting a build did not work in this
case.
- When there are many short-lived tasks, disk space was getting
checked very frequently. When the storage that is getting checked
is on an NFS server, that can lead to noticable traffic to the
server.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.
All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.
The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.
If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.
There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.
Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.
As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.
Fixes: [YOCTO #10668]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #10508]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This function is simplified by not trying to handle replacing the regex
and just compiling and using it for matching.
- Fix typo in logger output with undefined variable
- Fix pyflake errors
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.
We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.
[YOCTO #10220]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.
Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.
Also fix a number of pyflake errors.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.
Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.
[YOCTO #10220]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
toaster-bblayers.conf
Instead of updating conf/bblayers, here we update toaster-bblayers.conf
file. So extra effort to update bblayers.conf can be removed safely.
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|