| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The --clean flag to kgit-s2q exists to remove any fence post / sentinel
files that may be around from a previous run. That means --clean should
remove those files and then exit.
There were two issues with this, --clean was excuted too late in the
script AND it didn't exit after cleaning. This meant that if --clean
was issues before the tree was prepared, directories would be missing
and an error incorrectly thrown.
We move the clean operation to the top of the script and exit after
it is done and these issues are gone.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Once patching has completed the last patch applied is stored in the file
kgit-s2q.last. If for whatever reason patching is called again, patching
resumes from that point.
Without manually removing that file there is currently no way to remove
the sentinel file if you want patching to resume from the start of the
series.
We add a --clean option to remove the file and keep the internal
structure of tracking hidden.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Previous incarnations of the tools had a merge command that was
capable of merging feature branches into the build branch. This
support was removed to streamline processing of the tree.
Now that we have the .queue implementation, we can generate a
merge.queue from any merge commands found in .scc files and
allow the do_patch phase to take care of updating the tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Create a new command to merge a feature branch inside the current branch
when processing meta-series.
Signed-off-by: David Vincent <freesilicon@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Options with spacs around = signs will be droped/ignored by the
kernel.
The audit phase can detect this and warn the user:
% kconf_check --report -o .kernel-meta/cfg/ \
linux-qemux86-standard-build/.config `pwd` $cfgs
[errors (3): .kernel-meta/cfg/fragment_errors.txt
There are errors withing the config fragments.
% cat .kernel-meta/cfg/fragment_errors.txt
Warning: Ignoring "CONFIG_PARAVIRTT_CLOCK =y" -- spaces around equals are invalid
Warning: Ignoring "CONFIG_PARAVIRTTT_CLOCK = y" -- spaces around equals are invalid
Warning: Ignoring "CONFIG_PARAVIRTTTT_CLOCK= y" -- spaces around equals are invalid
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Commit 626ceac135fa66277c2fa53197be33cc9d4d7614 broke the error check in
process_file by adding in three lines that stomp on $? which print the
output file when verbose is set.
Move output file on verbose print to an elif after the error check.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The sum of the preprocessed input files are stored in a temporary
file. On exit, this file was not always being removed, which eventually
will clutter/fill /tmp.
We can remove it in the success path universally, and only keep it
around when explicitly asked to keep artifacts.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
A placeholder "unused.patch.queue" is created for runs that do
not have patches, or runs that have patch generation disabled.
That output directory was being created in /tmp, which is fine,
except that other outputs use dirname of the current queue and
hence follow it to /tmp.
We fix this by putting it in $outdir like the rest of the
artifacts, and it is safely away from any shared directory
(and subsequent permissions issues).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To provide useful error messages the tools dump pre-processed
files and messages to a temporary file. If multiple users are doing
builds, this means they either race, or can have permissions issues.
By creating the temporary file via mktemp, we avoid both issues.
(We also make sure to clean these up on exit, or /tmp will get
polluted quickly).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When processing input files per-branch and global patch queues are
generated. If the meta-data has not created any branches in the repo,
no branch specific queue is required.
The tools assumed that one is always valid, and hence would throw a
non-zero exit code and stop processing.
By testing for a named per-branch queue, we avoid this issue.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If there is an error in the processing of the input files, scc
should exit and inform the user.
scc is executed on a combined/preprocessed file and as a result
it doesn't have the granularity to see each input file individually.
Rather than moving preprocessing into scc (from spp), we can trap
the line number of the error and dump context around the line. This
gives the user a pointer to the input file and the specific line
that caused the problem.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
During a tree generation we must always make the branch active when we
see any kind of branch command. This ensures that any subsequent patches
are applied in the proper context.
Previously, only branch creation was changing the active branch, and
this mean that tree generation was not determinstic and relied on the
order of processing to generate a correct tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If a patch contains characters like ; or (), it can cause processing
errors in the patch queue.
By ensuring that we wrap the patch names in quotes, we avoid shell
processing and hence they are valid.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
During tree generation, the resumption of a process at
meta-series application was broken.
By fixing the working directory, we can now located (and
resume) meta series processing .. versus having to always
start from the beginning.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
s2q knows how to resume by looking at file in .git that logs the
last applied patch.
In situtations where a patch fails to apply (and is then resolved)
the series application won't resume properly since the last
applied patch was never logged.
We remove all the conditional logging of applied patches and simply
log each patch before it is applied. This ensures that we always
have the right thing tracked.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When a symbol is missing from the final .config, kconf_check can warn
the user (and point to the fragment that specified the option), but
figuring out why that option was dropped is left up to the end user.
This process normally involves going to the kernel source tree,
grepping for the config option and looking at the dependencies
(either in the source, or via menuconfig).
With Kconfiglib, we can load the Kconfig structure from the tree,
apply the active .config and provide diagnostics about the Kconfig
item, its dependencies and values of those dependencies. This allows
the user to fix most problems, without going to the source and
manually checking.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Add the script symbol_why to the kern-tools suite. This script takes
a .config, a kernel source directory and a configuration value as
inputs, and outputs a summary and/or conditional diagnostics on that
variable:
usage: symbol_why.py [-h] [-c DOTCONFIG] [-s KSRC] [-v] [--summary] [--prompt]
[--conditions] [--vars] [--value]
config ...
Kconfig symbol determination
positional arguments:
config configuration option to check
args <path to .config> <path to kernel source tree>
optional arguments:
-h, --help show this help message and exit
-c DOTCONFIG, --dotconfig DOTCONFIG
path to the .config to load (default: None)
-s KSRC, --ksrc KSRC path to the kernel source (default: None)
-v verbose (default: False)
--summary Show variable summary (default: False)
--prompt Show variable prompt (default: False)
--conditions Show config option dependencies (default: False)
--vars Show the dependent variables (default: False)
--value Show the config value (default: False)
Example when run from the kernel source directory:
% ARCH=x86 symbol_why.py --dotconfig=.config --ksrc=. BT_CMTP --summary
--conditions
Symbol BT_CMTP
Type : tristate
Value : "n"
User value : (no user value)
Visibility : "n"
Is choice item : false
Is defined : true
Is from env. : false
Is special : false
Prompts:
"CMTP protocol support" if BT_BREDR && ISDN_CAPI (value: "n")
Default values:
(no default values)
Selects:
(no selects)
Reverse (select-related) dependencies:
(no reverse dependencies)
Additional dependencies from enclosing menus and ifs:
NET (value: "y")
Locations: net/bluetooth/cmtp/Kconfig:1
Config 'BT_CMTP' has the following conditionals:
BT_BREDR && ISDN_CAPI (value: "n")
Dependency values are:
BT_BREDR [y] ISDN_CAPI [n]
And the user can now see details on the option, the final value, the
conditionals (dependencies) and what the value of the dependency
config items.
Hence, when an option is dropped for a .config, the information on
how to resolve the issue is provided to the user.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To provide better analysis of config items that don't make it into
the final .config, we can use Kconfiglib to load the .config and
parse the kernel's Kconfig files.
To avoid introducing a library dependency, we grab a copy of
Kconfiglib here and then make it available where ever the kern-tools
can be used.
Kconfiglib works standalone, so the capture makes sense, versus
looking for build system, or distro specific packaging.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The ability to sanitize and clean a destination repository is an
operation that many scripts may need. As such, we move this from
kgit-init to the main kgit script itself.
Any script that sources kgit, now has this available.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Recent changes to scc/spp process .scc files and generate meta-series
that are in a slightly different location. With this commit we can once
again process an entire kernel-cache and produce a branched and
populated git tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If the main kgit script is sourced (to provide common routines),
it shouldn't process command line arguments.
So we now detect if the script is sourced, or executed, and only
process command line arguments if it was executed.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Now that scc/spp and the other tools are modular (and smarter),
kgit-meta can be drastically simplified. It no longer needs to
worry about git, branches, kconfigs, or anything but taking
a meta series and applying it to a tree.
Currently this means branches and patches.
Additionally, kgit-meta is only used to generate a tree, so we
don't have to worry about it resuming or incorrect apply patches
when a developer is doing a build via a build system.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
kconf_check was coupled to the output of the other kern tools, in this
commit we break that coupling and allow fully standalone operation of
the script.
As such, kconf_check now only needs the following inputs:
- a linux kernel source tree
- a linux processed .config file
- a list of fragments or command line options
With these inputs a report is generated that finds common errors:
- redefined options
- dropped options
- invalid options
- repeated options (same fragment)
If the linux source directory includes meta data it will be detected
and used to classify the configuration options. This allows the reports
to be targetted to flagged options (versus reporting them all), but
meta data is completely optional and transparent to the end user.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
spp (the preprocessor) and scc (the compiler) are used to generate
a meta-series. This meta-series contains all the patches, commands,
etc, to construct a tree from scratch.
Other tools in the kernel build process then processed the meta-series
to trigger patching and configuration. This need to process the meta
series coupled the tools to spp/scc and made changing the build process
challenging.
With this change, spp/scc can now be invoked directly (without wrapper
scripts) and produce simple output files:
config.queue
patches.queue
These files are a list of configuration options or patches that should
be process to either configure .. or patch the tree. The tools used to
configure and patch the kernel are up to the end user (but in the
linux-yocto build process these are already defined).
In addition to these plain text queues, scc also produces a meta-series
like the existing meta-series. This can be used by other tools to
re-create a kernel tree from scratch, but is not part of the typical
patch and configuration workflow.
To implement these changes, spp/scc have been modified to load
"commands" at runtime. These commands are used to execute the contents
of .scc files (patch, kconf, mark, define, etc) in a sandbox. As such
the list of valid commands (and output) can be changed without changing
the scc tool itself. This allows multiple passes of the same input files
to produce different types of output.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We no longer need to carry any patches to merge_config, so we
update to the 4.7-rc6 variant.
A copy is still kept out of tree, since some users of the kern tools
do not want a complete clone of the kernel, just to get merge_config.sh
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To improve patching (and in particular re-patching) performance, we
store the last applied patch within the .git directory of a tree.
If this file is found, no resume detection or other processing happens,
it is simply taken as the fence post and patching resumes.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The code to find a meta directory was spread across many different
scripts. This obviously is a maintenance issue.
We centralize that code in the main kgit script and allow other
scripts to either source kgit and use the function, or call kgit --meta
from the command line.
Along with this new routine, we update the help message and add command
line processing to the script (since we can how pass options).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
With the re-work of the kern tools, we no longer need configme,
createme, patchme and updateme.
So we remove them from the repo and update the Makefile accordingly.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Once we've found a reason to push a patch, we should never stop
patching. Since if we top patching, then changes may be hidden
on ancestor branches.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
As part of processing configuration, patches and other operations the
paths of the files are normalized so they can be relocated from where
they were originally processed.
This is done by recording the directories that contain patches and
configuration, and then removing the longest possible path from each
element. Once that is done, we can search by their unique suffix across
any number of target directory structures.
The code that generated the path names, and the code that searches for
the longest prefix do not always agree on whether a trailing slash is
present. To cover all cases, we can simply try once with the slash
and once without.
When the relocation code misses the longest element, we end up with
arcane processing errors later in the tools since configs and patches
are not where they are expected.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When a .cfg is passed via the command line, the tools check to see if
a .scc file already include that fragment, and then avoid generating
a feature for that fragment (since it already has an explicit .scc
structure).
The tools then ensure that the .scc file is put into the meta-series
processing pipeline .. so the configuration fragment will be applied.
Unfortunately, this forced inclusion of the .scc file in the pipeline
can lead to duplicate processing, since there are other includes and
structure of the files in play.
To avoid the duplicate processing, if a .cfg file is already included
by a fragment, we trust the structure of the includes and don't force
it to be processed.
[YOCTO: #8486]
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When the auto-resume (resume point detection) was removed from the
processing of a meta-series, it ignored the fact that a single patch
series may in fact be processed a number of times.
Two layers patching a kernel will generate two different runs on the
same branch, which always start at patch one. This will obviously
break with duplicate patches.
To avoid this, we simply track the last patch applied, and explicitly
tell the patch scripts where to start. This gets us resume
functionality, without the overhead of resume point detection.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
On some machines (and kernel trees), it has been found that the branch
count may resolve to an empty string.
With the current patch/branch tests, this causes a patch application
error due to comparisons resolving with no right or left hand side.
To protect against this, we can combine the conditions and protect the
counts with quotes. This doesn't resolve the root cause, but is a
better, defensive condition.
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Previously the patching tools would consider both system and user
supplied patches in the same manner .. they are simply a series of
patches to be applied to a branch, and that the scripts should determine
where in the series to start (based on what is already on the branch).
This detection was causing a few problems:
- time consuming
- starting in the middle of a series when intermediate patches were
merged to a branch.
To solve both the performance and start detection, we instead simply
note the transition from system (i.e. already defined features and
series) and user/recipe supplied patches. When the transition is noted,
the system will start pushing ALL patches without doing autoresume
detection.
Control in keeping the series up to date is passed to the user, and
consistent behaviour/performance is achieved.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The existing meta directory detection was failing when multiple
files with .meta* or meta* where at the top level of a repository.
With this chance we can sucessfully detect, and use a meta directory
without needing to manually specify it.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We need to follow both parents in the history of a branch when dumping
the revision list. With merge commits, we may bring in a change from
multiple parent histories, and we should check both.
Without this patch, we may detect an invalid starting point to resume
a series.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To allow wrapping utilities to detect errors, if a meta series is not
generated an exit code of 1 should be returned.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
With the split of kernel meta from the main repo, and the ability to
run the tools in various different orders, we need to ensure that all
the scripts handle the detectiona and setting of the meta directory
the same way.
With this, we've sync'd the final two scripts to that way of handling
meta dirs.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
configme relies on kgit-meta to gather fragments for processing.
To allow configuration of an external meta-data repository and non
build system use cases, we need a single script with no dependencies
or complex command lines.
By moving the fragment gathering routines into configme, we have the
first step in this process complte.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The various scripts in the kern-tools have grown slightly different
meta directory detection routines over time. This means that we are
not always consistent, and the checks are not as robust as possible.
As part of this change, we have a single (which to avoid including
a common kgit script and coupling the tools .. is repeated) routine
for detecting the meta branch.
This detection logs the result in a new file ".metadir", which
subsequent scripts can find, and use, thereby avoid repeating the same
check many times.
We also enhance the check to look for a sentinel file in a proper meta
directory, to avoid false positives when an unexpected kernel process
leaves an uncommitted directory in the kernel dir.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We are allowed to restore a checkpoint multiple times, but detecting
if a checkpoint has already been restored depends on the proper meta
directory being checked.
In much the same way as other scripts, we should query the filesystem
for the right directory versus using a hardcoded default.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
In a similar way as commit 0768d697 [kgit-meta: dont run kgit-s2q for
non-leaf nodes], we can save even more processing time by not even
analysing and linking patches if we aren't on the leaf node of the
tree.
This early exit can save nearly 95% of the time required to "patch"
a tree when no changes are actually applied.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
Bruce Ashfield
David Vincent
George McCollister
Paul Gortmaker
Andreas Fenkart