| Age | Commit message (Collapse) | Author |
|---|
|
The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.
Note, we have to move the location of tinfiol's code to make it can be shutdown
correctly.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
-p, --prune
Before fetching, remove any remote-tracking references that no longer exist on the remote.
Fixed:
$ git push origin :test_branch
$ ./update.py
The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequantly. We only can update them via website without this patch,
which is not funny and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers to update_layer.py orderly can fix the problem, we
can get LAYERDEPENDS and LAYERRECOMMENDS info from tinfoil.
Not only new branch, but also existing branches may have the problem, because
BBFILE_COLLECTIONS maybe changed in the coming update, so we can't trust
database when the layer is going to be updated, for example, if there are 10
layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collection data from database, we need get
them from info again, so the code doesn't check whether it is a new branch or
not.
* Performance Improvement:
It should be faster than before in theory, since it ran update_layer.py
twice in the past, but now only once, I have tested it with 76 layers:
- Before: 4m25.912s, but only 30 layers were added, 46 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(4 * 60 + 25)/30*76/60 = 11.19m
- Now 8m5.315s, all the layers are added in the first run.
It improves from 11m to 8m.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Othewise it may cause troubles to the function who calls it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Fixed:
$ ./update.py -l meta-oe,not_existed
It should report error that "not_existed" doesn't exist, but it didn't, this was
because __in matches any of them.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Fixed:
- set LAYERDEPENDS_openembedded-layer = "core"
- $ "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core"
- Change LAYERDEPENDS_openembedded-layer = "foo"
- Run "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core and foo", this is
wrong, it should be "foo" only, this patch can fix the problem.
And also the existing checking should filter(required=required),
otherwise it can't work well when a layer is in both depends and
recommends, this can't happen in a normal case, but it would surprise the
user when this happens.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Drop a couple of items that got implemented recently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
In OE-Core master, HOSTTOOLS specifies a list of host-side tools to be
symlinked into a directory that is placed in a filtered path, so that
binaries used from the host are a controlled subset. This variable is
acted upon at configuration parse time and if any tools are missing,
parsing is terminated.
We're not actually running any tasks, so we won't need to call most of
the tools specified by HOSTTOOLS. There is one we definitely do need to
run (gcc) so keep that one, but the rest can be moved to
HOSTTOOLS_NONFATAL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we want to parse a configuration file (e.g. a distro conf file) then
we need convenient access to bitbake's conf parsing code, so create a
parse_conf() function to provide that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When the recipe search returns a single result, redirect to it
instead of showing the full result list view.
Part of the fix for [YOCTO #6618].
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The branch meta change made in b91b09a0aec11c0f8ee0cb99aa4ec81be9d23ddc
apparently requires this migration - not entirely sure why, but Django's
manage.py makemigrations wants to create it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Make minor (non-structural) improvements to the model, mostly for the
benefit of the admin interface:
* Set default ordering so that branches appear in the same order they do
in the user-facing drop-down
* Include the description in the default string representation (so you
know which branch name matches with which Yocto Project version, in
the OE index).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We should always be using four spaces.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We can't decode UTF-8 characters byte-by-byte, as soon as we hit a
character that's more than one byte long then we'll fail. Use a reader
object to do it properly. This fixes parsing current meta-angstrom on
master. At the same time, specify errors="surrogateescape" to avoid the
update process dying at this point in case of characters that aren't
valid UTF-8.
Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's
very similar code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Layers that aren't published shouldn't be visible via the API. (We don't
need to apply that filter to recipes, machines or distros though since a
layer's content won't automatically be indexed unless it has been
published).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
On the main layer index page we want openembedded-core to appear before
meta-oe - naturally openembedded-core is the layer everyone starts from.
We already have an "index_preference" column that we use for search
results, so just sort by that here before sorting by name.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
In the layer detail page, show counts on all but the updates tab (i.e.
for recipes, machines, appends, classes, and distros).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Since some recent infrastructure changes, "/cgit.cgi" should no longer
be part of cgit.openembedded.org URLs, apparently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This wasn't documented anywhere so I've added it here. I'm not sure
it'll get used outside of the live OE index but it's at least useful for
anyone else who needs to debug it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Fix the transaction handling code to work with Django 1.8 in most of the
tools scripts. (Some of these are no longer used, but still serve as
examples of how to import data and update the database.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
I neglected to update this at the time I updated the README recently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
A minor optimisation - it's not going to change between layers so get it
in the branch loop.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
* Setting of the object has to be before the try: or otherwise the
finally block can get called if that doesn't succeed with the result
that the layerconfparser object won't have a value, which will
trigger an exception
* We shouldn't be using the config_data object after shutting down
tinfoil, so avoid doing that
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Fix a couple of problems with shutting down tinfoil:
1) tinfoil.shutdown() wasn't being called on error because it wasn't
inside a finally: section, thus on error the script hung with bitbake
master (since the tinfoil2 changes) - this is almost certainly a bug
in bitbake but let's handle it here anyway.
2) We need to check whether the tinfoil object actually has a shutdown
attribute since we still want to support updating for older branches
where tinfoil didn't have a shutdown() method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
You need to use a single = not == when filtering in Django.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We might as well show the latest tested point release.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Remove ; and spaces when setting variables in calls.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Don't assume remote name is origin, run `git remote` to get the remote
name. When checking the remote, detect the branch as well, that way the
layerindex will work if the remote branch name and local branch name do
not match.
Note that this currently only supports one remote.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
It's a little crude and certainly not optimal performance-wise, but we
can support querying for recipes that inherit a particular class without
too much trouble. This allows you to add "inherits:cmake" to the query
and have it return only recipes that inherit the cmake class. You can
use more than one inherits: item to filter down to recipes that inherit
all of the specified classes.
Note: this does not otherwise change the behaviour of specifying
multiple words - all of the words other than those that start with
"inherits:" are treated as part of a single phrase that will be searched
for - not separate keywords.
Fixes [YOCTO #9879].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If you specified printerr=False we were referring to the output variable
that hadn't been set. Looks like I broke this back in 2013 in
93ce26f21cdbbd8a645792359cde87acf05144d7.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Add an action to duplicate a Branch object, along with all of the
LayerBranches (and LayerMaintainers and LayerDependencies) underneath
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
update_layer may create the layerbranch record (e.g. for a new branch)
so we should be looking for it after that has been run. Also, we cannot
assume that last_rev will get set because a layer might fail to fetch,
so take that into account as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If the layer failed to fetch, we shouldn't be attempting to update its
contents. (This got broken when we split the update script into two in
c64e4c57a9158c8ae5e49f526c3ff87950d3a94e).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Sometimes people put values that aren't URLs into the HOMEPAGE variable.
If that's the case, then we should not turn that value into a link which
will be invalid.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Ensure recent contributions from Mark and Liam are reflected in the
authors list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
For some repo URLs we can automatically determine the values of all of
the other fields - e.g. for github or git.openembedded.org, and we've
been doing that for a while. However if someone submits a URL for some
other site we don't know what type of interface it uses, and usually
the submitter leaves the fields blank so it falls to the layer index
maintainer to set the values, and then you have to remember what the
correct URL format is which is awkward especially for gitweb.
In order to fix this, add a select field to the form which allows
specifying which type of interface is being used. At the moment only
cgit, gitweb, gitlab and "(custom)" (i.e. the current behaviour) are
supported. This is not a real field but activates javascript code
that sets the other fields and enables/disables the controls.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Improves slightly on 3155206e54413f72df3b3b41280eafd332a58ba4 by doing
an exact match on name and showing that first - now when you search for
"git" you really do get the git recipe first in the list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Usage of itertools.chain() that was introduced in
3155206e54413f72df3b3b41280eafd332a58ba4 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
Robert Yang
Paul Eggleton
Alex Franco
Aníbal Limón
Liam R. Howlett