Age | Commit message (Collapse) | Author |
|
Invalid shell characters in patch names were fixed via:
[
commit 9a3995ee8daabf37e92e1b51b133cf8582d85809
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Tue Oct 18 11:13:43 2016 -0400
patch: make patches with shell-unsafe characters valid
If a patch contains characters like ; or (), it can cause processing
errors in the patch queue.
By ensuring that we wrap the patch names in quotes, we avoid shell
processing and hence they are valid.
]
A similar problem can occur when inhibited patches are processed
(i.e. included foo.scc nopatch), since the inhibited names are
logged to the output files.
To fix this issue, we wrap these patch names in quotes to avoid them
being mistaken for shell functions.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The sum of the preprocessed input files are stored in a temporary
file. On exit, this file was not always being removed, which eventually
will clutter/fill /tmp.
We can remove it in the success path universally, and only keep it
around when explicitly asked to keep artifacts.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
A placeholder "unused.patch.queue" is created for runs that do
not have patches, or runs that have patch generation disabled.
That output directory was being created in /tmp, which is fine,
except that other outputs use dirname of the current queue and
hence follow it to /tmp.
We fix this by putting it in $outdir like the rest of the
artifacts, and it is safely away from any shared directory
(and subsequent permissions issues).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To provide useful error messages the tools dump pre-processed
files and messages to a temporary file. If multiple users are doing
builds, this means they either race, or can have permissions issues.
By creating the temporary file via mktemp, we avoid both issues.
(We also make sure to clean these up on exit, or /tmp will get
polluted quickly).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When processing input files per-branch and global patch queues are
generated. If the meta-data has not created any branches in the repo,
no branch specific queue is required.
The tools assumed that one is always valid, and hence would throw a
non-zero exit code and stop processing.
By testing for a named per-branch queue, we avoid this issue.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If there is an error in the processing of the input files, scc
should exit and inform the user.
scc is executed on a combined/preprocessed file and as a result
it doesn't have the granularity to see each input file individually.
Rather than moving preprocessing into scc (from spp), we can trap
the line number of the error and dump context around the line. This
gives the user a pointer to the input file and the specific line
that caused the problem.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
During a tree generation we must always make the branch active when we
see any kind of branch command. This ensures that any subsequent patches
are applied in the proper context.
Previously, only branch creation was changing the active branch, and
this mean that tree generation was not determinstic and relied on the
order of processing to generate a correct tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If a patch contains characters like ; or (), it can cause processing
errors in the patch queue.
By ensuring that we wrap the patch names in quotes, we avoid shell
processing and hence they are valid.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
During tree generation, the resumption of a process at
meta-series application was broken.
By fixing the working directory, we can now located (and
resume) meta series processing .. versus having to always
start from the beginning.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
s2q knows how to resume by looking at file in .git that logs the
last applied patch.
In situtations where a patch fails to apply (and is then resolved)
the series application won't resume properly since the last
applied patch was never logged.
We remove all the conditional logging of applied patches and simply
log each patch before it is applied. This ensures that we always
have the right thing tracked.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When a symbol is missing from the final .config, kconf_check can warn
the user (and point to the fragment that specified the option), but
figuring out why that option was dropped is left up to the end user.
This process normally involves going to the kernel source tree,
grepping for the config option and looking at the dependencies
(either in the source, or via menuconfig).
With Kconfiglib, we can load the Kconfig structure from the tree,
apply the active .config and provide diagnostics about the Kconfig
item, its dependencies and values of those dependencies. This allows
the user to fix most problems, without going to the source and
manually checking.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Add the script symbol_why to the kern-tools suite. This script takes
a .config, a kernel source directory and a configuration value as
inputs, and outputs a summary and/or conditional diagnostics on that
variable:
usage: symbol_why.py [-h] [-c DOTCONFIG] [-s KSRC] [-v] [--summary] [--prompt]
[--conditions] [--vars] [--value]
config ...
Kconfig symbol determination
positional arguments:
config configuration option to check
args <path to .config> <path to kernel source tree>
optional arguments:
-h, --help show this help message and exit
-c DOTCONFIG, --dotconfig DOTCONFIG
path to the .config to load (default: None)
-s KSRC, --ksrc KSRC path to the kernel source (default: None)
-v verbose (default: False)
--summary Show variable summary (default: False)
--prompt Show variable prompt (default: False)
--conditions Show config option dependencies (default: False)
--vars Show the dependent variables (default: False)
--value Show the config value (default: False)
Example when run from the kernel source directory:
% ARCH=x86 symbol_why.py --dotconfig=.config --ksrc=. BT_CMTP --summary
--conditions
Symbol BT_CMTP
Type : tristate
Value : "n"
User value : (no user value)
Visibility : "n"
Is choice item : false
Is defined : true
Is from env. : false
Is special : false
Prompts:
"CMTP protocol support" if BT_BREDR && ISDN_CAPI (value: "n")
Default values:
(no default values)
Selects:
(no selects)
Reverse (select-related) dependencies:
(no reverse dependencies)
Additional dependencies from enclosing menus and ifs:
NET (value: "y")
Locations: net/bluetooth/cmtp/Kconfig:1
Config 'BT_CMTP' has the following conditionals:
BT_BREDR && ISDN_CAPI (value: "n")
Dependency values are:
BT_BREDR [y] ISDN_CAPI [n]
And the user can now see details on the option, the final value, the
conditionals (dependencies) and what the value of the dependency
config items.
Hence, when an option is dropped for a .config, the information on
how to resolve the issue is provided to the user.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To provide better analysis of config items that don't make it into
the final .config, we can use Kconfiglib to load the .config and
parse the kernel's Kconfig files.
To avoid introducing a library dependency, we grab a copy of
Kconfiglib here and then make it available where ever the kern-tools
can be used.
Kconfiglib works standalone, so the capture makes sense, versus
looking for build system, or distro specific packaging.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The ability to sanitize and clean a destination repository is an
operation that many scripts may need. As such, we move this from
kgit-init to the main kgit script itself.
Any script that sources kgit, now has this available.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Recent changes to scc/spp process .scc files and generate meta-series
that are in a slightly different location. With this commit we can once
again process an entire kernel-cache and produce a branched and
populated git tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
If the main kgit script is sourced (to provide common routines),
it shouldn't process command line arguments.
So we now detect if the script is sourced, or executed, and only
process command line arguments if it was executed.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Now that scc/spp and the other tools are modular (and smarter),
kgit-meta can be drastically simplified. It no longer needs to
worry about git, branches, kconfigs, or anything but taking
a meta series and applying it to a tree.
Currently this means branches and patches.
Additionally, kgit-meta is only used to generate a tree, so we
don't have to worry about it resuming or incorrect apply patches
when a developer is doing a build via a build system.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
kconf_check was coupled to the output of the other kern tools, in this
commit we break that coupling and allow fully standalone operation of
the script.
As such, kconf_check now only needs the following inputs:
- a linux kernel source tree
- a linux processed .config file
- a list of fragments or command line options
With these inputs a report is generated that finds common errors:
- redefined options
- dropped options
- invalid options
- repeated options (same fragment)
If the linux source directory includes meta data it will be detected
and used to classify the configuration options. This allows the reports
to be targetted to flagged options (versus reporting them all), but
meta data is completely optional and transparent to the end user.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
spp (the preprocessor) and scc (the compiler) are used to generate
a meta-series. This meta-series contains all the patches, commands,
etc, to construct a tree from scratch.
Other tools in the kernel build process then processed the meta-series
to trigger patching and configuration. This need to process the meta
series coupled the tools to spp/scc and made changing the build process
challenging.
With this change, spp/scc can now be invoked directly (without wrapper
scripts) and produce simple output files:
config.queue
patches.queue
These files are a list of configuration options or patches that should
be process to either configure .. or patch the tree. The tools used to
configure and patch the kernel are up to the end user (but in the
linux-yocto build process these are already defined).
In addition to these plain text queues, scc also produces a meta-series
like the existing meta-series. This can be used by other tools to
re-create a kernel tree from scratch, but is not part of the typical
patch and configuration workflow.
To implement these changes, spp/scc have been modified to load
"commands" at runtime. These commands are used to execute the contents
of .scc files (patch, kconf, mark, define, etc) in a sandbox. As such
the list of valid commands (and output) can be changed without changing
the scc tool itself. This allows multiple passes of the same input files
to produce different types of output.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We no longer need to carry any patches to merge_config, so we
update to the 4.7-rc6 variant.
A copy is still kept out of tree, since some users of the kern tools
do not want a complete clone of the kernel, just to get merge_config.sh
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To improve patching (and in particular re-patching) performance, we
store the last applied patch within the .git directory of a tree.
If this file is found, no resume detection or other processing happens,
it is simply taken as the fence post and patching resumes.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The code to find a meta directory was spread across many different
scripts. This obviously is a maintenance issue.
We centralize that code in the main kgit script and allow other
scripts to either source kgit and use the function, or call kgit --meta
from the command line.
Along with this new routine, we update the help message and add command
line processing to the script (since we can how pass options).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
With the re-work of the kern tools, we no longer need configme,
createme, patchme and updateme.
So we remove them from the repo and update the Makefile accordingly.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Once we've found a reason to push a patch, we should never stop
patching. Since if we top patching, then changes may be hidden
on ancestor branches.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
As part of processing configuration, patches and other operations the
paths of the files are normalized so they can be relocated from where
they were originally processed.
This is done by recording the directories that contain patches and
configuration, and then removing the longest possible path from each
element. Once that is done, we can search by their unique suffix across
any number of target directory structures.
The code that generated the path names, and the code that searches for
the longest prefix do not always agree on whether a trailing slash is
present. To cover all cases, we can simply try once with the slash
and once without.
When the relocation code misses the longest element, we end up with
arcane processing errors later in the tools since configs and patches
are not where they are expected.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Signed-off-by: Andreas Fenkart <afenkart@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When a .cfg is passed via the command line, the tools check to see if
a .scc file already include that fragment, and then avoid generating
a feature for that fragment (since it already has an explicit .scc
structure).
The tools then ensure that the .scc file is put into the meta-series
processing pipeline .. so the configuration fragment will be applied.
Unfortunately, this forced inclusion of the .scc file in the pipeline
can lead to duplicate processing, since there are other includes and
structure of the files in play.
To avoid the duplicate processing, if a .cfg file is already included
by a fragment, we trust the structure of the includes and don't force
it to be processed.
[YOCTO: #8486]
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When the auto-resume (resume point detection) was removed from the
processing of a meta-series, it ignored the fact that a single patch
series may in fact be processed a number of times.
Two layers patching a kernel will generate two different runs on the
same branch, which always start at patch one. This will obviously
break with duplicate patches.
To avoid this, we simply track the last patch applied, and explicitly
tell the patch scripts where to start. This gets us resume
functionality, without the overhead of resume point detection.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
On some machines (and kernel trees), it has been found that the branch
count may resolve to an empty string.
With the current patch/branch tests, this causes a patch application
error due to comparisons resolving with no right or left hand side.
To protect against this, we can combine the conditions and protect the
counts with quotes. This doesn't resolve the root cause, but is a
better, defensive condition.
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Previously the patching tools would consider both system and user
supplied patches in the same manner .. they are simply a series of
patches to be applied to a branch, and that the scripts should determine
where in the series to start (based on what is already on the branch).
This detection was causing a few problems:
- time consuming
- starting in the middle of a series when intermediate patches were
merged to a branch.
To solve both the performance and start detection, we instead simply
note the transition from system (i.e. already defined features and
series) and user/recipe supplied patches. When the transition is noted,
the system will start pushing ALL patches without doing autoresume
detection.
Control in keeping the series up to date is passed to the user, and
consistent behaviour/performance is achieved.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The existing meta directory detection was failing when multiple
files with .meta* or meta* where at the top level of a repository.
With this chance we can sucessfully detect, and use a meta directory
without needing to manually specify it.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We need to follow both parents in the history of a branch when dumping
the revision list. With merge commits, we may bring in a change from
multiple parent histories, and we should check both.
Without this patch, we may detect an invalid starting point to resume
a series.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To allow wrapping utilities to detect errors, if a meta series is not
generated an exit code of 1 should be returned.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
With the split of kernel meta from the main repo, and the ability to
run the tools in various different orders, we need to ensure that all
the scripts handle the detectiona and setting of the meta directory
the same way.
With this, we've sync'd the final two scripts to that way of handling
meta dirs.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
configme relies on kgit-meta to gather fragments for processing.
To allow configuration of an external meta-data repository and non
build system use cases, we need a single script with no dependencies
or complex command lines.
By moving the fragment gathering routines into configme, we have the
first step in this process complte.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
The various scripts in the kern-tools have grown slightly different
meta directory detection routines over time. This means that we are
not always consistent, and the checks are not as robust as possible.
As part of this change, we have a single (which to avoid including
a common kgit script and coupling the tools .. is repeated) routine
for detecting the meta branch.
This detection logs the result in a new file ".metadir", which
subsequent scripts can find, and use, thereby avoid repeating the same
check many times.
We also enhance the check to look for a sentinel file in a proper meta
directory, to avoid false positives when an unexpected kernel process
leaves an uncommitted directory in the kernel dir.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
We are allowed to restore a checkpoint multiple times, but detecting
if a checkpoint has already been restored depends on the proper meta
directory being checked.
In much the same way as other scripts, we should query the filesystem
for the right directory versus using a hardcoded default.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
In a similar way as commit 0768d697 [kgit-meta: dont run kgit-s2q for
non-leaf nodes], we can save even more processing time by not even
analysing and linking patches if we aren't on the leaf node of the
tree.
This early exit can save nearly 95% of the time required to "patch"
a tree when no changes are actually applied.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
Much the same way that we save dropped configuration for later
reporting, we make a copy of non-hardware specifications with this
change.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
After a mismatch (a kernel configuration option that doesn't make the
final .config) is found, reporting tools may need access to the details
without needing to mine down into the configuration audit directory
structure.
To faciliate this, we make a copy of mismatch.cfg (and any other files
that we deem reportable) to the top level meta directory (along side
the meta-series).
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When changing branches it used to be a purely addititive operation
if a branch existed. This is no longer the case, so we shouldn't
preserve the existig series file, but should instead clear the series
file, and let the patch routines add to it if required.
This was triggered by the patch optimization changes, that no longer
run do_patch if a leaf/final branch is not being processed.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To ensure that all patche are considered and applied to the tree, we
pass --gen to kgit-meta when we know we are generating a tree from a
kernel meta data repository.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
When a history complete treegen is being performed, the meta data respository
is imported and its history grafted onto the kernel tree. When we operate in
this mode, the tree is then checkpoint restored before the generation of the
tree is performed.
A checkpointed restored tree has meta data in .$META, not $META. So we switch
to using the dot directory when this is detected.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|
|
To speed up the processing of a meta-series, we can skip patch
validation on anything but the leaf branch, since we do not push
and propagate changes from sub branches to the final/BSP branch.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
|