aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2023-12-29layerindex: improve updates for actual_branchtimo/update-button-actual-branchTim Orling
Some layers now have one branch with many supported LAYERSERIES_COMPAT. If this branch name does not match one of the stable releases, LayerBranches might not have been created. When actual_branch is set, it is only set in a LayerBranch object. We previously could not update (create) a stable branch with actual_branch except manually in the admin interface. Add --force-create option to be used in conjunction with --actual-branch (which already requires --branch) in the update.py script. This tells the script to ignore the fact that no layerbranch exists already. Add --actual-branch to update_layer.py so that we can create (and more importantly checkout) an actual_branch for the given stable --branch. Update utils.py to allow checking out of actual_branch when a LayerBranch does not yet exist. While we are at it, ensure that any Branch that is marked as no update will be skipped even with --force-create. The main reason that a Branch has updates disabled is because the bitbake or python syntax has changed enough to cause exceptions. This script can now be run with: ./layerindex/update.py \ --layer meta-weird-one \ --branch kirkstone \ --actual-branch=nonstandard \ --force-create Which will attempt to create a meta-weird-one:kirkstone layerbranch checked out at the 'nonstandard' branch from that layer's git repo. This allows layerindex admins to at least populate the database without tedious creation of layerbranches in the admin interface. Helps make the "branch mapping" actually work and be useful: [YOCTO #8008] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29layerindex: add Update Layer UI featureTim Orling
* Add an "Update Layer" button to the layer detail view. - This allows a user that is a member of is_staff to trigger an update of the current layer (for the current branch) * Add an "Update Layer" button to the reviewdetail view - This allows a user that is a member of is_staff and has publish_layer permissions to trigger an update attempt of the layer under review (even in the un-published state) * The update is run as a task with Celery NOTE: You must have the RABBITMQ_ and DATABASE_ credentials set correctly in the docker/settings.py file or set via environment variables or you will get authentication errors talking to layersdb or layersrabbit containers. [YOCTO #12484] layerindex/views.py: add update_layer_view layerindex/urls.py: add update_layer_view layerindex/urls_branch.py: add update_layer_view templates/layerindex/reviewdetail.html: add Update Layer button templates/layerindex/detail.html: add Update Layer button templates/layerindex: add updatelayer.html TODO: While the update is happening, the AJAX rendering of the update.log is showing the b'' characters and not adding any new lines. If you go back to the same task view afterwards, the log is rendered as expected. TODO: After the update is completed, it would be nice to have a button to return you to the page from where you called the "Update Layer". Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29layerindex: Add actual_branch to forms and viewsTim Orling
For layers which do not follow standard branch names (including the inclusive naming move away from "master" to "main") we have the actual_branch field set in a LayerBranch object. Previously this was only exposed via the admin interface. Allow layer maintainers (including upon submitting a new layer) to set the 'Actual branch' in the web UI. Add a check to make sure the actual_branch is a valid branch name using 'git check-ref-format --branch <actual_branch>' since we are not using full refs. [YOCTO #8008] NOTE: Only existing LayerBranches will be editable. A new layer can be submitted with a different branch for "master", but only the "master" LayerBranch will be created. Further changes to the update.py script will be needed to make creation of new stable branches with an actual_branch possible. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29dockersetup.py: enable production Let's EncryptTim Orling
The '--staging' argument to certbot has now been changed to '--test-cert'. We previously only allowed using the dockersetup.py tool to create Staging environment certs, which are still marked as invalid by browsers. Add a '--letsencrypt-production' knob to allow for valid, trusted certs to be created. If they already exist in the workspace and have not expired, re-use them (to avoid hitting rate limits). Continue to '--force-renewal' for staging certs. NOTE: If you have previously created staging certs in your workspace, you will want to clean docker/certs before creating production certs for the same domain. Certbot will not overwrite those staging certs and the newly created ones will not be in the path passed in by dockersetup.py. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29docker/settings.py: fix RABBIT_BROKER TypeErrorTim Orling
Within the layersapp container, if you try to run layerindex/update.py: File "/opt/layerindex/settings.py", line 280, in <module> RABBIT_BROKER = 'amqp://' + os.getenv('RABBITMQ_DEFAULT_USER') + ':' + os.getenv('RABBITMQ_DEFAULT_PASS') + '@layersrabbit:5672/' TypeError: can only concatenate str (not "NoneType") to str Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29layerindex/views.py: fix parse_view typoTim Orling
The Branch Comparison view would throw an error because of a parser_view instead of parse_view typo. [YOCTO #15332] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-12-29requirements.txt: bump all to latestTim Orling
Pin mysqlclient as greater than 2.1.1 changes behavior Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-22utils.py: fix unclosed (Tim Orling
File "/opt/layerindex/layerindex/update.py", line 525 failed_layers[branch].append('%s: Failed to add since LAYERDEPENDS [%s ...] is not SyntaxError: '(' was never closed Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-17update.py: display first unsatisfied LAYERDEPENDSTim Orling
Make errors like this more useful: ERROR: Issues found on branch nanbield: meta-luneos: Failed to add since LAYERDEPENDS is not satisfied meta-luneui: Failed to add since LAYERDEPENDS is not satisfied In this case, meta-luneos depends on meta-luneui, but we cannot create a 'nanbield' layer branch, because meta-luneui LAYERDEPENDS on qt6-layer collection, which currently has no 'nanbield' branch. Use next(iter(value['deps'])) to get the first element in the OrderedDict Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-17update.py: show branch if missing conf/layer.confTim Orling
Currently, the error message is something like: ERROR: conf/layer.conf not found for layer meta-doom - is subdirectory set correctly? This is because in this case meta-doom has a 'langdale' branch without a conf/layer.conf Another example is: ERROR: conf/layer.conf not found for layer meta-st-stm32mp - is subdirectory set correctly? This is because meta-st-stm32mp has a master branch, but it only contains a README.md telling you to use the stable branches (e.g. nanbield). Make it more obvious what the source of the error is by also displaying the branch that was being attempted. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-17layerindex/models.py: fix Inactive-Upstream checkTim Orling
The regex for Upstream-Status had (\w+), but this does not include hyphen/dash. Explicitly add it to the pattern. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-16utils.py: get dependent layers at same branchTim Orling
For both meta-poky/meta-yocto and meta-xilinx/meta-xilinx-core we have a situation where the "collections" stayed the same ("yocto" and "xilinx" respectively) but the layer/layerbranch changed. Without the "branch" argument to get_dependency_layers, we were always defaulting to the older layer which first defined the "collection". Instead, add an option to use "branch" to filter on the expected LayerBranch object. Keep the old behavior just in case someone depends upon it. [YOCTO #15221] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-16update_layer.py: checkout dep layers as wellTim Orling
We have never been checking out dependent layers at the same release/branch. With the introduction of 'addpylib', this became obvious due to parsing errors. Ensure that known LayerDependency objects are checked out at the expected branch/release. Since openembedded-core has already been handled elsewhere, we skip it. [YOCTO #15236] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-16update.py: fix updates_enabled for newbranchTim Orling
When a new release is run for the first time, no LayerBranch exists yet and the update_enabled cannot be queried. The unintended result is that all layers are skipped. Move the update_enabled check into the code path where the layerbranch already exists. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-16layerindex/tools: add mark_yp_compatible_layers.pyTim Orling
Add a script which can either mark one --layer --branch or use data in a --from-file to mark multiple layer:branch objects as Yocto Project Compatible. The --from-file is a json file is compatible with or which can be generated by: yocto-autobuilder-helper/scripts/list-yp-compatible-layers.py [YOCTO #15093] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-11-16Add SECURITY.mdTim Orling
Add the generic Yocto Project tools SECURITY.md as a first step. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-19bootstrap: fix typo in minified file nameTim Orling
Fixes dropdown menus so they are functional again. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-19update.py: skip unless layerbranch.updates_enabledTim Orling
If layerbranch.updates_enabled is false, skip the update. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-19models.py: add updates_enabled to LayerBranchTim Orling
Especially since LTS branches have become more popular as the only active branch, we need to be able to turn off updates on a LayerBranch basis rather than the current Layer or Branch heavier hammers. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-16Chart.js: upgrade from 2.9.3 to 2.9.4Tim Orling
https://github.com/chartjs/Chart.js/releases/tag/v2.9.4 CVE: CVE-2020-7746 https://nvd.nist.gov/vuln/detail/CVE-2020-7746 Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-16Chart.js: add version to filenamesTim Orling
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-16bootstrap: add version to filenamesTim Orling
To make it easier to quickly see what version we are using, add the version to the filenames. Also, use minified flavor in templates/base.html Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-16jquery: upgrade 3.4.1 -> 3.7.1Tim Orling
Use minified version in templates/base.html Fixes some vulernabilities: https://nvd.nist.gov/vuln/search/results?adv_search=true&isCpeNameSearch=true&query=cpe%3A2.3%3Aa%3Ajquery%3Ajquery%3A3.4.1%3A*%3A*%3A*%3A*%3Anode.js%3A*%3A* CVE: CVE-2020-23064 CVE: CVE-2020-11022 CVE: CVE-2020-11023 For full changelog see: https://github.com/jquery/jquery/compare/3.4.1...3.7.1 License-Update: Copyright OpenJS Foundation (from JS Foundation) Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-06docker/settings.py: add CSRF_TRUSTED_ORIGINSTim Orling
In addition to ALLOWED_HOSTS, we now must have CSRF_TRUSTED_ORIGINS defined. This variable requires the scheme (http:// or https://). Like ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS is a list of strings, with one entry for each host which is trusted for POST requests. https://docs.djangoproject.com/en/4.2/ref/settings/#csrf-trusted-origins Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05password_validation.py: fix deprecated ugettextTim Orling
Use django.utils.translation.gettext instead Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05requirements.txt: update dependenciesTim Orling
Use results of: pipenv install pipenv update pip freeze > requirements-freeze.txt Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05requirements.txt: upgrade all to latestTim Orling
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05Dockerfile: drop python3-pilTim Orling
This prevents upgrading Pillow in requirements.txt Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05Refactor usage of django.conf.urlsTim Orling
django.conf.urls.url() was removed in Django 4.0: https://docs.djangoproject.com/en/4.2/releases/4.0/#features-removed-in-4-0 Replace all usage with django.urls.re_path() Replace all django.conf.urls imports with equivalent django.urls modules Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05bootstrap_pagination: disable via commentTim Orling
Wrap the is_pagination usage of bootstrap_pagination with comment/endcomment to avoid incompatible module. We will need to re-write the pagination code. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05Drop django-bootstrap-pagination: incompatibleTim Orling
Upstream django-bootstrap-pagination is no longer maintained and is incompatible with Django 4.x. https://github.com/staticdev/django-pagination-bootstrap was archived on June 26, 2023. ugettext was removed in Django 4.0: https://docs.djangoproject.com/en/4.2/releases/4.0/#features-removed-in-4-0 Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05docker-compose.yml: bump mariadb from 10.3 to 10.4Tim Orling
Django 4.2 requires a minimum of 10.4 https://docs.djangoproject.com/en/4.2/releases/4.2/#dropped-support-for-mariadb-10-3 Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05requirements.txt: upgrade Django 3.2 -> 4.2Tim Orling
Update to 4.2 LTS release https://docs.djangoproject.com/en/4.2/releases/4.2/ https://docs.djangoproject.com/en/4.1/releases/4.1/ https://docs.djangoproject.com/en/4.0/releases/4.0/ Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05requirements: bump django-simple-captcha to 0.5.20Tim Orling
https://raw.githubusercontent.com/mbi/django-simple-captcha/master/CHANGES Version 0.5.20 -------------- * Still support Django 3.2 (#222, thanks @petrklus) Version 0.5.19 -------------- * SECURITY ISSUE: reset the random seed after an image was generated (#221, thanks @ibuler) Version 0.5.18 -------------- * Fix some typos in documentation (#210, thanks @stweil) * Test against Django 4.2 * Stopped testing Django < 3.2 * BaseCaptchaTextInput should set autocomplete=off on the hashkey HiddenInput (#201, thanks @eerotal) * Test against Django 4.2a * Fix some deprecation warnings in Pillow 9.2+ * Removed old unused conditional imports * Format code with pre-commit and black Version 0.5.17 -------------- * English translation created from Spanish ones (#209, thanks @tpazderka) Version 0.5.16 -------------- * Adds a migration missing from 0.5.15 (#208, thanks @atodorov) Version 0.5.15 -------------- * Updated test matrix, drop tests against Python3.6, test against Python3.9 * Remove dependency on six * Test against Django 4.0a1 * Test with Python 3.10 (Django 3.2 and Django 4.0) * Remove warning for django 3.2 (#206, thanks @MiStErLu) Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05dockersetup.py: databasefile zstd compressionTim Orling
zstd compression and decompression are significantly faster than gzip. zstd is also "splittable" and is streaming/real-time friendly. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-10-05build(deps): bump pillow from 9.3.0 to 10.0.1dependabot[bot]
Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.3.0 to 10.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/9.3.0...10.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-09-29build(deps): bump gitpython from 3.1.32 to 3.1.37Tim Orling
This release contains another security fix that further improves validation of symbolic references and thus properly fixes this CVE: https://github.com/advisories/GHSA-cwvm-v4w8-q58c (CVE-2023-41040). https://github.com/gitpython-developers/GitPython/blob/main/doc/source/changes.rst Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-09-29layerindex/urls.py: fix submit/thanks/ urlTim Orling
The url returned by the reverse lookup ends in /submit/thanks/ but the regex was improperly searching for a string without the terminating "/". Thank you to Pawel Zalewski for reporting the 404 error. Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-08-25build(deps): bump gitpython from 3.1.30 to 3.1.32timo/mailing-list-patchesdependabot[bot]
Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-08-25build(deps): bump pillow from 9.2.0 to 9.3.0dependabot[bot]
Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.2.0 to 9.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/9.2.0...9.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2023-08-25requirements.txt: bump gitpython and pillowChee Yang Lee
bump gitpython and pillow to version with fixes for these CVEs pillow: CVE-2022-22817 CVE-2022-24303 CVE-2022-45198 gitpython: CVE-2022-24439 Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
2022-12-20layerindex/utils.py: Add to baseconfig=True for bb.parse.handle()Robert Yang
Bitbake's api has been changed via: afb8478d3 parse: Add support for addpylib conf file directive and BB_GLOBAL_PYMODULES The conf file won't be parsed without baseconfig=True: bb.parse.ParseError: ParseError at /path/to/oe-core/meta/conf/layer.conf:132: unparsed line: 'addpylib ${LAYERDIR}/lib oe' Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16global: replace deprecated distutilsTim Orling
Replace distutils.version.LooseVersion with pkg_resource.parse_version https://majornetwork.net/2021/05/comparing-version-numbers-in-python/ [YOCTO #14990] Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16Dockerfile: switch to ubuntu:jammy for >python3.8Michael Halstead
ubuntu:jammy provides python 3.10.6 which supports current bitbake. Dropped unneeded python2 packages. Moved locale generation high to reduce error messages during image building. Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16utils.py: Fix for is_commit_ancestor()Robert Yang
The runcmd() would print an "ERROR" on failure which causes confusion since the failure is expected on old branches, so subprocess.getstatusoutput to fix the problem. Minor rewording. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16admin.py: Remove has_delete_permission()Robert Yang
The following commit has added on_delete=models.CASCADE to the ForeignKey: === commit 2d526f9b0d363f3e442abc97ef9bd9fab37ee5e4 Author: Amber Elliot <amber.n.elliot@intel.com> Date: Thu Jun 25 15:57:34 2020 -0700 Updating models and imports for Django 2.2 upgrade. === The on_delete=models.CASCADE will remove the objects which reference the removed objects, so there will be errors when removing a LayerBranch with admin permissions: === Cannot delete layer branch Deleting the selected layer branch would result in deleting related objects, but your account doesn't have permission to delete the following types of objects: Append distro === This is because has_delete_permission() always return False, remove it to use Django's implementation to fix the problem. Minor rewording. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16utils.py: Use get_layer_var() to get BBFILE_COLLECTIONSRobert Yang
The commit 38e6288c7dad579518b0de2b6664f23be58889b6 has added oe-core to BBLAYERS by default, and then there will be multiple BBFILE_COLLECTIONS which causes update.py failure to update layers (most of the dependencies or recommends are missing), use get_layer_var() to get BBFILE_COLLECTIONS to fix the problem since it can ignore the extra oe-core from BBFILE_COLLECTIONS. Fix minor typos. Minor rewording. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-12-16Dockerfile: add captcha dependenciesMichael Halstead
The current version of django-simple-captcha requires libz and freetype development libraries. Refactor to apply. Fix typo. Fix previous mixed use of space vs tab. Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-04-28layerindex/utils.py: ignore 'core' in BBFILES_COLLECTIONSTim Orling
Many layers append BBFILE_COLLECTIONS and therefore have 'core <layer>' During update.py, this means we are likely not handling the collection we expect: WARNING: /opt/workdir/git___git_openembedded_org_meta-openembedded/meta-oe: multiple collections found, handling first one (core) only BBFILE_COLLECTIONS = "core" Signed-off-by: Tim Orling <tim.orling@konsulko.com>
2022-04-28recipe{desc,parse}.py: BB_ENV_PASSTHROUGH_ADDITIONSTim Orling
ERROR: Variable BB_ENV_EXTRAWHITE has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS ERROR: Variable BB_ENV_EXTRAWHITE from the shell environment has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS ERROR: Exiting to allow enviroment variables to be corrected Replace BB_ENV_EXTRAWHITE with new variable BB_ENV_PASSTHROUGH_ADDITIONS In order to be backward compatible with older branches, we must first check for the presence of the bitbake commit which implemented the variable name change, using layerindex.utils.is_commit_ancestor(). Signed-off-by: Tim Orling <tim.orling@konsulko.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-06-07 10:57:26 +0000