diff --git a/dev-support/docs/README.md b/dev-support/docs/README.md new file mode 100644 index 0000000000000..610014cadefbc --- /dev/null +++ b/dev-support/docs/README.md @@ -0,0 +1,30 @@ + + +# Overview + +Yetus helps community driven software projects improve their contribution and release processes by providing: + +* A robust system for automatically checking new contributions against a variety of community accepted requirements +* The means to document a well defined supported interface for downstream projects +* Tooling to help release managers generate release documentation based on the information provided by community issue trackers and source repositories + +# Yetus Precommit + +The Yetus Precommit Patch Tester allows projects to codify their patch acceptance criteria and then evaluate incoming contributions prior to review by a committer. + +* Take a quick look at [our glossary of terms](precommit-glossary.md) to ensure you are familiar with the ASF and Maven jargon we'll use as terminology specific to this project. +* For an overview of Yetus' philosophy on testing contributions and how evaluation is performed, see our [overview](precommit-architecture.md). +* To get started on your project, including an explanation of what we'll expect in a runtime environment and what optional utilities we'll leverage, read through the [basic usage guide](precommit-basic.md). +* If your project has advanced requirements such as module relationships not expressed in Maven, special profiles, or a need on os-specific prerequisites not managed by Maven then you'll need to use our [advanced usage guide](precommit-advanced.md). diff --git a/dev-support/docs/precommit-advanced.md b/dev-support/docs/precommit-advanced.md new file mode 100644 index 0000000000000..32865d31684dd --- /dev/null +++ b/dev-support/docs/precommit-advanced.md @@ -0,0 +1,243 @@ + + +test-patch +========== + +* [Docker Support](#Docker_Support) +* [Maven Specific](#Maven_Specific) +* [Ant Specific](#Ant_Specific) +* [Plug-ins](#Plug-ins) +* [Configuring for Other Projects](#Configuring_for_Other_Projects) +* [Important Variables](#Important_Variables) + +# Docker Support + +By default, test-patch runs in the same shell where it was launched. It can alternatively use Docker to launch itself into a container. This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java. + +The `--docker` parameter tells test-patch to run in Docker mode. The `--dockerfile` parameter allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary binaries and tooling needed to run the test. However be aware that test-patch will copy this file and append its necessary hooks to re-launch itself prior to executing docker. + +NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX) or C:\Users (Windows) as the base and patchprocess directories (specified by the --basedir and --patch-dir options respectively), because automatically mountable directories are limited to them. See [the Docker documentation](https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume). + +Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --jenkins mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. + +# Maven Specific + +## Command Arguments + +test-patch always passes --batch-mode to maven to force it into non-interactive mode. Additionally, some tests will also force -fae in order to get all of messages/errors during that mode. It *does not* pass -DskipTests. Additional arguments should be handled via the personality. + +## Test Profile + +By default, test-patch will pass -Ptest-patch to Maven. This will allow you to configure special actions that should only happen when running underneath test-patch. + +# Ant Specific + +## Command Arguments + +test-patch always passes -noinput to Ant. This force ant to be non-interactive. + +# Plug-ins + +test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. + +## Test Plug-ins + +Every test plugin must have one line in order to be recognized: + +```bash +add_plugin +``` + +This function call registers the `pluginname` so that test-patch knows that it exists. This plug-in name also acts as the key to the custom functions that you can define. For example: + +```bash +function pluginname_filefilter +``` + +This function gets called for every file that a patch may contain. This allows the plug-in author to determine if this plug-in should be called, what files it might need to analyze, etc. + +Similarly, there are other functions that may be defined during the test-patch run: + +* pluginname_postcheckout + - executed prior to the patch being applied but after the git repository is setup. This is useful for any early error checking that might need to be done before any heavier work. + +* pluginname_preapply + - executed prior to the patch being applied. This is useful for any "before"-type data collection for later comparisons. + +* pluginname_postapply + - executed after the patch has been applied. This is useful for any "after"-type data collection. + +* pluginname_postinstall + - executed after the mvn install test has been done. If any tests require the Maven repository to be up-to-date with the contents of the patch, this is the place. + +* pluginname_tests + - executed after the unit tests have completed. + +If the plug-in has some specific options, one can use following functions: + +* pluginname_usage + + - executed when the help message is displayed. This is used to display the plug-in specific options for the user. + +* pluginname_parse_args + + - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. + + HINT: It is recommended to make the pluginname relatively small, 10 characters at the most. Otherwise, the ASCII output table may be skewed. + +## Bug System Plug-ins + +Similar to tests, the ability to add support for bug tracking systems is also handled via a plug-in mechanism. + +* pluginname_usage + + - executed when the help message is displayed. This is used to display the plug-in specific options for the user. + +* pluginname\_parse\_args + + - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. + + +* pluginname\_locate\_patch + + - Given input from the user, download the patch if possible. + +* pluginname\_determine\_branch + + - Using any heuristics available, return the branch to process, if possible. + +* pluginname\_determine\_issue + + - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table. + +* pluginname_writecomment + + - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately. + +* pluginname\_linecomments + + - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments. + +* pluginname_finalreport + + - Write the final result table to the bug system. + +# Configuring for Other Projects + +It is impossible for any general framework to be predictive about what types of special rules any given project may have, especially when it comes to ordering and Maven profiles. In order to direct test-patch to do the correct action, a project `personality` should be added that enacts these custom rules. + +A personality consists of two functions. One that determines which test types to run and another that allows a project to dictate ordering rules, flags, and profiles on a per-module, per-test run. + +There can be only **one** of each personality function defined. + +## Test Determination + +The `personality_file_tests` function determines which tests to turn on based upon the file name. It is relatively simple. For example, to turn on a full suite of tests for Java files: + +```bash +function personality_file_tests +{ + local filename=$1 + + if [[ ${filename} =~ \.java$ ]]; then + add_test findbugs + add_test javac + add_test javadoc + add_test mvninstall + add_test unit + fi + +} +``` + +The `add_test` function is used to activate the standard tests. Additional plug-ins (such as checkstyle), will get queried on their own. + +## Module & Profile Determination + +Once the tests are determined, it is now time to pick which [modules](precommit-glossary.md#genericoutside-definitions) should get used. That's the job of the `personality_modules` function. + +```bash +function personality_modules +{ + + clear_personality_queue + +... + + personality_enqueue_module + +} +``` + +It takes exactly two parameters `repostatus` and `testtype`. + +The `repostatus` parameter tells the `personality` function exactly what state the source repository is in. It can only be in one of two states: `branch` or `patch`. `branch` means the patch has not been applied. The `patch` state is after the patch has been applied. + +The `testtype` state tells the personality exactly which test is about to be executed. + +In order to communicate back to test-patch, there are two functions for the personality to use. + +The first is `clear_personality_queue`. This removes the previous test's configuration so that a new module queue may be built. Custom personality_modules will almost always want to do this as the first action. + +The second is `personality_enqueue_module`. This function takes two parameters. The first parameter is the name of the module to add to this test's queue. The second parameter is an option list of additional flags to pass to Maven when processing it. `personality_enqueue_module` may be called as many times as necessary for your project. + + NOTE: A module name of . signifies the root of the repository. + +For example, let's say your project uses a special configuration to skip unit tests (-DskipTests). Running unit tests during a javadoc build isn't very useful and wastes a lot of time. We can write a simple personality check to disable the unit tests: + + +```bash +function personality_modules +{ + local repostatus=$1 + local testtype=$2 + + if [[ ${testtype} == 'javadoc' ]]; then + personality_enqueue_module . -DskipTests + return + fi + ... + +``` + +This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool. + + + +# Important Variables + +There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information. + +* BUILD\_NATIVE will be set to true if the system has requested that non-JVM-based code be built (e.g., JNI or other compiled C code). Under Jenkins, this is always true. + +* BUILDTOOL specifies which tool is currently being used to drive compilation. Additionally, many build tools define xyz\_ARGS to pass on to the build tool command line. (e.g., MAVEN\_ARGS if maven is in use). Projects may set this in their personality. NOTE: today, only one build tool at a time is supported. This may change in the future. + +* CHANGED\_FILES is a list of all files that appear to be added, deleted, or modified in the patch. + +* CHANGED\_UNFILTERED\_MODULES is a list of all modules that house all of the CHANGED\_FILES. Be aware that the root of the source tree is reported as '.'. + +* CHANGED\_MODULES reports which modules that appear to have source code in them. + +* HOW\_TO\_CONTRIBUTE should be a URL that points to a project's on-boarding documentation for new users. Currently, it is used to suggest a review of patch naming guidelines. Since this should be project specific information, it is useful to set in a project's personality. + +* JIRA\_ISSUE\_RE is to help test-patch when talking to JIRA. It helps determine if the given project is appropriate for the given JIRA issue. There are similar variables for GITHUB. + +* MODULE and other MODULE\_\* are arrays that contain which modules, the status, etc, to be operated upon. These should be treated as read-only by plug-ins. + +* PATCH\_BRANCH\_DEFAULT is the name of the branch in the git repo that is considered the master. This is useful to set in personalities. + +* PATCH\_DIR is the name of the temporary directory that houses test-patch artifacts (such as logs and the patch file itself) + +* TEST\_PARALLEL if parallel unit tests have been requested. Project personalities are responsible for actually enabling or ignoring the request. TEST\_THREADS is the number of threads that have been requested to run in parallel. diff --git a/dev-support/docs/precommit-architecture.md b/dev-support/docs/precommit-architecture.md new file mode 100644 index 0000000000000..cd527ae0e0c59 --- /dev/null +++ b/dev-support/docs/precommit-architecture.md @@ -0,0 +1,97 @@ + + +# Some Philosophy + +* Everyone's time is valuable. The quicker contributors can get feedback and iterate, the more likely and faster their contribution will get checked in. A committer should be able to focus on the core issues of a contribution rather than details that can be determined automatically. + +* Precommit checks should be fast. There is no value in testing parts of the source tree that are not immediately impacted by a change. Unit testing is the target. They are not a replacement for full builds or integration tests. + +* Many open source projects have a desire to have this capability. Why not generalize a solution? + +* In many build systems (especially with maven), a modular design has been picked. Why not leverage that design to make checks faster? + +* Projects that use the same language will, with a high degree of certainty, benefit from the same types of checks. + +* Portability matters. Tooling should be as operating system and language agnostic as possible. + +# Phases + +test-patch works effectively under several different phases: + +## Setup + +This is where test-patch configures and validates the environment. Some things done in this phase: + +* Defaults +* Parameter handling +* Importing plugins and personalities +* Docker container launching +* Re-exec support +* Patch file downloading +* git repository management (fresh pull, branch switching, etc) + +## Post-checkout + +Checks done here are *fatal*. + +This acts as a verification of all of the setup parts and is the final place to short-cut the full test cycle. The most significant built-in check done here is verifying the patch file is a valid. + +## Pre-apply + +This is where the 'before' work is handled. Some things that typically get checked in this phase: + +* The first pass of files and modules that will get patched +* Validation and information gathering of the source tree pre-patch +* Author checks +* Check for modified unit tests + +## Patch is Applied + +The patch gets applied. Then a second pass to determine which modules and files have been changed in order to handle any modules that might have added or moved. + +## Post-apply + +Now that the patch has been applied, many of the same checks performed in the Pre-apply step are done again to build an 'after' picture. + +## Post-install + +Some tests only work correctly when the repo is up-to-date. So +mvn install is run to update the local repo and we enter this phase. Some example tests performed here: + +* javadoc +* Findbugs +* Maven eclipse integration still works + +## Unit Tests + +Since unit tests are generally the slowest part of the precommit process, they are run last. At this point, all the prerequisites to running them should be in place and ready to go. + +## Reporting + +Finally, the results are reported to the screen and, optionally, to JIRA and/or whatever bug system has been configured. + +# Test Flow + +The basic workflow for many of the sub-items in individual phases are: + +1. print a header, so the end user knows that something is happening +1. verify if the test is needed. If so, continue on. Otherwise, return success and let the next part of the phase execute. +1. Ask the personality about what modules and what flags should get used +1. Execute maven (or some other build tool) in the given modules with the given flags. Log the output and record the time and result code. +1. Do any extra work as appropriate (diffs, counts, etc) and either accept the status and message given by the maven run or change the vote, message, log file, etc, based upon this extra work. +1. Add the outcome(s) to the report generator + +As one can see, the modules list is one of the key inputs into what actually gets executed. As a result, projects must full flexibility in either adding, modifying, or even removing modules from the test list. If a personality removes the entire list of modules, then that test should just be ignored. + diff --git a/dev-support/docs/precommit-basic.md b/dev-support/docs/precommit-basic.md new file mode 100644 index 0000000000000..07af634fc509c --- /dev/null +++ b/dev-support/docs/precommit-basic.md @@ -0,0 +1,222 @@ + + +test-patch +========== + +* [Purpose](#Purpose) +* [Pre-requisites](#Pre-requisites) +* [Basic Usage](#Basic_Usage) +* [Build Tool](#Build_Tool) +* [Providing Patch Files](#Providing_Patch_Files) +* [Project-Specific Capabilities](#Project-Specific_Capabilities) +* [MultiJDK](#MultiJDK) +* [Docker](#Docker) + +# Purpose + +As part of Hadoop's commit process, all patches to the source base go through a precommit test that does some (relatively) light checking to make sure the proposed change does not break unit tests and/or passes some other prerequisites such as code formatting guidelines. This is meant as a preliminary check for committers so that the basic patch is in a known state and for contributors to know if they have followed the project's guidelines. This check, called test-patch, may also be used by individual developers to verify a patch prior to sending to the Hadoop QA systems. + +Other projects have adopted a similar methodology after seeing great success in the Hadoop model. Some have even gone as far as forking Hadoop's precommit code and modifying it to meet their project's needs. + +This is a modification to Hadoop's version of test-patch so that we may bring together all of these forks under a common code base to help the community as a whole. + + +# Pre-requisites + +test-patch has the following requirements: + +* Ant- or Maven-based project (and ant/maven installed) +* git-based project (and git 1.7.3 or higher installed) +* bash v3.2 or higher +* findbugs 3.x installed +* shellcheck installed, preferably 0.3.6 or higher +* pylint installed +* GNU diff +* GNU patch +* POSIX awk +* POSIX grep +* POSIX sed +* curl +* file command +* smart-apply-patch.sh (included!) + +Maven plugins requirements: + +* Apache RAT +* Apache FindBugs + +Optional: + +* JIRA-based issue tracking +* GitHub-based issue tracking + +The locations of these files are (mostly) assumed to be in the file path, but may be overridden via command line options. For Solaris and Solaris-like operating systems, the default location for the POSIX binaries is in /usr/xpg4/bin and the default location for the GNU binaries is /usr/gnu/bin. + + +# Basic Usage + +This command will execute basic patch testing against a patch file stored in "filename": + +```bash +$ cd +$ dev-support/test-patch.sh --dirty-workspace --project=projectname +``` + +The `--dirty-workspace` flag tells test-patch that the repository is not clean and it is ok to continue. By default, unit tests are not run since they may take a significant amount of time. + +To do turn them on, we need to provide the --run-tests option: + + +```bash +$ cd +$ dev-support/test-patch.sh --dirty-workspace --run-tests +``` + +This is the same command, but now runs the unit tests. + +A typical configuration is to have two repositories. One with the code you are working on and another, clean repository. This means you can: + +```bash +$ cd +$ git diff master > /tmp/patchfile +$ cd ../ +$ /dev-support/test-patch.sh --basedir= --resetrepo /tmp/patchfile +``` + +We used two new options here. --basedir sets the location of the repository to use for testing. --resetrepo tells test patch that it can go into **destructive** mode. Destructive mode will wipe out any changes made to that repository, so use it with care! + +After the tests have run, there is a directory that contains all of the test-patch related artifacts. This is generally referred to as the patchprocess directory. By default, test-patch tries to make something off of /tmp to contain this content. Using the `--patch-dir` option, one can specify exactly which directory to use. This is helpful for automated precommit testing so that Jenkins or other automated workflow system knows where to look to gather up the output. + +For example: + +```bash +$ test-patch.sh --jenkins --patch-dir=${WORKSPACE}/patchprocess --basedir=${WORKSPACE}/source ${WORKSPACE}/patchfile +``` + +... will trigger test-patch to run in fully automated Jenkins mode, using ${WORKSPACE}/patchprocess as its scratch space, ${WORKSPACE}/source as the source repository, and ${WORKSPACE}/patchfile as the name of the patch to test against. + +# Build Tool + +Out of the box, test-patch is built to use maven. But what if the project is built using something else, such as ant? + +```bash +$ test-patch.sh (other options) --build-tool=ant +``` + +will tell test-patch to use ant instead of maven to drive the project. + +# Providing Patch Files + +## JIRA + +It is a fairly common practice within the Apache community to use Apache's JIRA instance to store potential patches. As a result, test-patch supports providing just a JIRA issue number. test-patch will find the *last* attachment, download it, then process it. + +For example: + +```bash +$ test-patch.sh (other options) HADOOP-9905 +``` + +... will process the patch file associated with this JIRA issue. + +If the Apache JIRA system is not in use, then override options may be provided on the command line to point to a different JIRA instance. + +```bash +$ test-patch.sh --jira-issue-re='^PROJECT-[0-9]+$' --jira-base-url='https://example.com/jira' PROJECT-90 +``` + +... will process the patch file attached to PROJECT-90 on the JIRA instance located on the example.com server. + +## GITHUB + +A new practice within the ASF is to use a service such as GitHub and its Pull Request (PR) feature. test-patch supports many forms of providing PR support. + +```bash +$ test-patch.sh --github-repo=apache/pig 99 +``` + +or + +```bash +$ test-patch.sh https://github.com/apache/pig/pulls/99 +``` + +or + +```bash +$ test-patch.sh https://github.com/apache/pig/pulls/99.patch +``` + +... will process PR #99 on the apache/pig repo. + +## Generic URLs + +Luckily, test-patch supports provide ways to provide unified diffs via URLs. + +For example: + +```bash +$ test-patch.sh (other options) https://example.com/webserver/file.patch +``` + +... will download and process the file.patch from the example.com webserver. + +# Project-specific Capabilities + +Due to the extensible nature of the system, test-patch allows for projects to define project-specific rules which we call personalities. (How to build those rules is covered elsewhere.) There are two ways to specify which personality to use: + +## Direct Method + +```bash +$ test-patch.sh (other options) --personality=(filename) +``` + +This tells test-patch to use the personality in the given file. + +## Project Method + +However, test-patch can detect if it is a personality that is in its "personality" directory based upon the project name: + +```bash +$ test-patch.sh (other options) --project=(project) +``` + +# MultiJDK + +For many projects, it is useful to test Java code against multiple versions of JDKs at the same time. test-patch can do this with the --multijdkdirs option: + +```bash +$ test-patch.sh (other options) --multijdkdirs="/j/d/k/1,/j/d/k/2" +``` + +Not all Java tests support this mode, but those that do will now run their tests with all of the given versions of Java consecutively (e.g., javac--the Java compliation test). Tests that do not support MultiJDK mode (e.g., checkstyle, mvn install) will use JAVA\_HOME. + +NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode. If JAVA\_HOME is in the list, it will be moved to the end. + +# Docker + +test-patch also has a mode to utilize Docker: + +```bash +$ test-patch.sh (other options) --docker +``` + +This will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile, see the advanced guide. + +## In Closing + +test-patch has many other features and command line options for the basic user. Many of these are self-explanatory. To see the list of options, run test-patch.sh without any options or with --help. + + diff --git a/dev-support/docs/precommit-glossary.md b/dev-support/docs/precommit-glossary.md new file mode 100644 index 0000000000000..ca3d15fa79bdf --- /dev/null +++ b/dev-support/docs/precommit-glossary.md @@ -0,0 +1,37 @@ + + +# Glossary + +## Generic/outside definitions: + +Apache's [new contributor documentation](https://community.apache.org/contributors/) and Maven's [glossary](https://maven.apache.org/glossary.html) are great places to start if you are new to Apache or Maven. + +* Module + + Almost the same meaning as "sub-project" on Maven. + +## test-patch specific + +* Personality + + A chunk of shell code that tells test-patch this particular project's needs and special requirements + +* Plug-ins + + Shell code snippets that define new, not built-in test types. + +* Precommit + + An automated process that verifies a patch is "good" prior to a committer looking at it. diff --git a/dev-support/docs/releasedocmaker.md b/dev-support/docs/releasedocmaker.md new file mode 100644 index 0000000000000..b39de2e64c9db --- /dev/null +++ b/dev-support/docs/releasedocmaker.md @@ -0,0 +1,115 @@ + + +releasedocmaker +=============== + +* [Purpose](#Purpose) +* [Basic Usage](#Basic_Usage) +* [Changing the Header](#Changing_the_Header) +* [Multiple Versions](#Multiple_Versions) +* [Unreleased Dates](#Unreleased_Dates) +* [Lint Mode](#Lint_Mode) + +# Purpose + +Building changelog information in a form that is human digestible but still containing as much useful information is difficult. Many attempts over the years have resulted in a variety of methods that projects use to solve this problem: + +* JIRA-generated release notes from the "Release Notes" button +* Manually modified CHANGES file +* Processing git log information + +All of these methods have their pros and cons. Some have issues with accuracy. Some have issues with lack of details. None of these methods seem to cover all of the needs of many projects and are full of potential pitfalls. + +In order to solve these problems, releasedocmaker was written to automatically generate a changelog and release notes by querying Apache's JIRA instance. + +# Basic Usage + +Minimally, the name of the JIRA project and a version registered in JIRA must be provided: + +```bash +$ releasedocmaker.py --project (project) --version (version) +``` + +This will query Apache JIRA, generating two files in a directory named after the given version in an extended markdown format which can be processed by both mvn site and GitHub. + +* CHANGES.(version).md + +This is similar to the JIRA "Release Notes" button but is in tabular format and includes the priority, component, reporter, and contributor fields. It also highlights Incompatible Changes so that readers know what to look out for when upgrading. The top of the file also includes the date that the version was marked as released in JIRA. + + +* RELEASENOTES.(version).md + +If your JIRA project supports the release note field, this will contain any JIRA mentioned in the CHANGES log that is either an incompatible change or has a release note associated with it. If your JIRA project does not support the release notes field, this will be the description field. + +For example, to build the release documentation for HBase v1.2.0... + +```bash +$ releasedocmaker.py --project HBASE --version 1.2.0 +``` + +... will create a 1.2.0 directory and inside that directory will be CHANGES.1.2.0.md and RELEASENOTES.1.2.0.md . + + +# Changing the Header + +By default, it will use a header that matches the project name. But that is kind of ugly and the case may be wrong. Luckily, the title can be changed: + +```bash +$ releasedocmaker.py --project HBASE --version 1.2.0 --projecttitle "Apache HBase" +``` + +Now instead of "HBASE", it will use "Apache HBASE" for some titles and headers. + +# Multiple Versions + +The script can also generate multiple versions at once, by + +```bash +$ releasedocmaker.py --project HBASE --version 1.0.0 --version 1.2.0 +``` + +This will create the files for versions 1.0.0 and versions 1.2.0 in their own directories. + +But what if the version numbers are not known? releasedocmaker can also generate version data based upon ranges: + +```bash +$ releasedocmaker.py --project HBASE --version 1.0.0 --version 1.2.0 --range +``` + +In this form, releasedocmaker will query JIRA, discover all versions that alphabetically appear to be between 1.0.0 and 1.2.0, inclusive, and generate all of the relative release documents. This is especially useful when bootstrapping an existing project. + +# Unreleased Dates + +For released versions, releasedocmaker will pull the date of the release from JIRA. However, for unreleased versions it marks the release as "Unreleased". This can be inconvenient when actually building a release and wanting to include it inside the source package. + +The --usetoday option can be used to signify that instead of using Unreleased, releasedocmaker should use today's date. + +```bash +$ releasedocmaker.py --project HBASE --version 1.0.0 --usetoday +``` + +After using this option and release, don't forget to change JIRA's release date to match! + +# Lint Mode + +In order to ensure proper formatting while using mvn site, releasedocmaker puts in periods (.) for fields that are empty or unassigned. This can be unsightly and not proper for any given project. There are also other things, such as missing release notes for incompatible changes, that are less than desirable. + +In order to help release managers from having to scan through potentially large documents, releasedocmaker features a lint mode, triggered via --lint: + +```bash +$ releasedocmaker.py --project HBASE --version 1.0.0 --lint +``` + +This will do the normal JIRA querying, looking for items it considers problematic. It will print the information to the screen and then exit with either success or failure, depending upon if any issues were discovered. diff --git a/dev-support/findHangingTest.sh b/dev-support/findHangingTest.sh old mode 100644 new mode 100755 diff --git a/dev-support/personality/flink.sh b/dev-support/personality/flink.sh new file mode 100755 index 0000000000000..0c068635d3ce2 --- /dev/null +++ b/dev-support/personality/flink.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#shellcheck disable=SC2034 +PATCH_BRANCH_DEFAULT=master +#shellcheck disable=SC2034 +JIRA_ISSUE_RE='^FLINK-[0-9]+$' +#shellcheck disable=SC2034 +GITHUB_REPO="apache/flink" +#shellcheck disable=SC2034 +HOW_TO_CONTRIBUTE="" + +add_plugin flinklib + +function fliblib_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.java$ + || ${filename} =~ \.scala$ + || ${filename} =~ pom.xml$ ]]; then + add_test flinklib + fi +} + +function flinklib_count +{ + find "${BASEDIR}" \ + | ${GREP} "/lib/" \ + | ${GREP} -v "_qa_workdir" \ + | wc -l +} + +function flinklib_preapply +{ + start_clock + big_console_header "${PATCH_BRANCH} flink library dependencies" + + verify_needed_test flinklib + if [[ $? == 0 ]]; then + echo "Patch does not need flinklib testing." + return 0 + fi + + pushd "${BASEDIR}" >/dev/null + echo_and_redirect "${PATCH_DIR}/branch-flinklib-root.txt" \ + "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch + if [[ $? != 0 ]]; then + add_vote_table -1 flinklib "Unable to determine flink libs in ${PATCH_BRANCH}." + fi + FLINK_PRE_LIB_FILES=$(flinklib_count) + popd >/dev/null +} + +function flinklib_postapply +{ + start_clock + big_console_header "Patch flink library dependencies" + + verify_needed_test flinklib + if [[ $? == 0 ]]; then + echo "Patch does not need flinklib testing." + return 0 + fi + + pushd "${BASEDIR}" >/dev/null + echo_and_redirect "${PATCH_DIR}/patch-flinklib-root.txt" \ + "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch + FLINK_POST_LIB_FILES=$(flinklib_count) + popd >/dev/null + + + if [[ "${FLINK_POST_LIB_FILES}" -gt "${FLINK_PRE_LIB_FILES}" ]]; then + add_vote_table -1 flinklib "Patch increases lib folder dependencies from " \ + "${FLINK_PRE_LIB_FILES} to ${FLINK_POST_LIB_FILES}" + return 1 + elif [[ "${FLINK_POST_LIB_FILES}" -eq "${FLINK_PRE_LIB_FILES}" ]]; then + add_vote_table 0 flinklib "Patch did not change lib dependencies" \ + " (still ${FLINK_PRE_LIB_FILES})" + else + add_vote_table +1 flinklib "Patch decreases lib folder dependencies by " \ + "$((FLINK_PRE_LIB_FILES-FLINK_POST_LIB_FILES))." + fi + return 0 +} + +function personality_modules +{ + local repostatus=$1 + local testtype=$2 + local extra="" + + yetus_debug "Personality: ${repostatus} ${testtype}" + + clear_personality_queue + + case ${testtype} in + mvninstall) + if [[ ${repostatus} == branch ]]; then + personality_enqueue_module . -DskipTests + return + fi + return + ;; + asflicense) + # this is very fast and provides the full path if we do it from + # the root of the source + personality_enqueue_module . + return + ;; + unit) + if [[ ${TEST_PARALLEL} == "true" ]] ; then + extra="-Pparallel-tests" + if [[ -n ${TEST_THREADS:-} ]]; then + extra="${extra} -DtestsThreadCount=${TEST_THREADS}" + fi + fi + ;; + *) + extra="-DskipTests" + ;; + esac + + for module in ${CHANGED_MODULES}; do + # shellcheck disable=SC2086 + personality_enqueue_module ${module} ${extra} + done +} + +function personality_file_tests +{ + local filename=$1 + + yetus_debug "Using personality_file_tests, but calling the built-in:" + builtin_personality_file_tests "${1}" + + if [[ ${filename} =~ \.scala$ ]]; then + add_test unit + fi +} diff --git a/dev-support/personality/hadoop.sh b/dev-support/personality/hadoop.sh new file mode 100755 index 0000000000000..f3c64127c00b2 --- /dev/null +++ b/dev-support/personality/hadoop.sh @@ -0,0 +1,350 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Override these to match Apache Hadoop's requirements + +#shellcheck disable=SC2034 +PATCH_BRANCH_DEFAULT=trunk +#shellcheck disable=SC2034 +HOW_TO_CONTRIBUTE="https://wiki.apache.org/hadoop/HowToContribute" +#shellcheck disable=SC2034 +JIRA_ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$' +#shellcheck disable=SC2034 +GITHUB_REPO="apache/hadoop" +#shellcheck disable=SC2034 +PYLINT_OPTIONS="--indent-string=' '" + +HADOOP_MODULES="" + +function hadoop_module_manipulation +{ + local startingmodules=${1:-normal} + local module + local hdfs_modules + local ordered_modules + local tools_modules + local passed_modules + local flags + + yetus_debug "hmm in: ${startingmodules}" + + if [[ ${startingmodules} = normal ]]; then + startingmodules=${CHANGED_MODULES} + elif [[ ${startingmodules} = union ]]; then + startingmodules=${CHANGED_UNION_MODULES} + fi + + yetus_debug "hmm expanded to: ${startingmodules}" + + if [[ ${startingmodules} = "." ]]; then + yetus_debug "hmm shortcut since ." + HADOOP_MODULES=. + return + fi + + # ${startingmodules} is already sorted and uniq'd. + # let's remove child modules if we're going to + # touch their parent. + passed_modules=${startingmodules} + for module in ${startingmodules}; do + yetus_debug "Stripping ${module}" + # shellcheck disable=SC2086 + passed_modules=$(echo ${passed_modules} | tr ' ' '\n' | ${GREP} -v ${module}/ ) + done + + yetus_debug "hmm pre-ordering: ${startingmodules}" + + # yarn will almost always be after common in the sort order + # so really just need to make sure that common comes before + # everything else and tools comes last + + for module in ${passed_modules}; do + yetus_debug "Personality ordering ${module}" + if [[ ${module} = "." ]]; then + HADOOP_MODULES=. + break + fi + + if [[ ${module} = hadoop-hdfs-project* ]]; then + hdfs_modules="${hdfs_modules} ${module}" + elif [[ ${module} = hadoop-common-project/hadoop-common + || ${module} = hadoop-common-project ]]; then + ordered_modules="${ordered_modules} ${module}" + elif [[ ${module} = hadoop-tools* ]]; then + tools_modules="${tools_modules} ${module}" + else + ordered_modules="${ordered_modules} ${module}" + fi + done + + HADOOP_MODULES="${ordered_modules} ${hdfs_modules} ${tools_modules}" + + yetus_debug "hmm out: ${HADOOP_MODULES}" +} + +function hadoop_unittest_prereqs +{ + local need_common=0 + local building_common=0 + local module + local flags + local fn + + for module in ${HADOOP_MODULES}; do + if [[ ${module} = hadoop-hdfs-project* ]]; then + need_common=1 + elif [[ ${module} = hadoop-common-project/hadoop-common + || ${module} = hadoop-common-project ]]; then + building_common=1 + fi + done + + if [[ ${need_common} -eq 1 + && ${building_common} -eq 0 ]]; then + echo "unit test pre-reqs:" + module="hadoop-common-project/hadoop-common" + fn=$(module_file_fragment "${module}") + flags=$(hadoop_native_flags) + pushd "${BASEDIR}/${module}" >/dev/null + # shellcheck disable=SC2086 + echo_and_redirect "${PATCH_DIR}/maven-unit-prereq-${fn}-install.txt" \ + "${MAVEN}" "${MAVEN_ARGS[@]}" install -DskipTests ${flags} + popd >/dev/null + fi +} + +function hadoop_native_flags +{ + + if [[ ${BUILD_NATIVE} != true ]]; then + return + fi + + # Based upon HADOOP-11937 + # + # Some notes: + # + # - getting fuse to compile on anything but Linux + # is always tricky. + # - Darwin assumes homebrew is in use. + # - HADOOP-12027 required for bzip2 on OS X. + # - bzip2 is broken in lots of places. + # e.g, HADOOP-12027 for OS X. so no -Drequire.bzip2 + # + + # current build servers are pretty limited in + # what they support + if [[ ${JENKINS} = true + && ${DOCKERSUPPORT} = false ]]; then + # shellcheck disable=SC2086 + echo -Pnative \ + -Drequire.snappy -Drequire.openssl -Drequire.fuse \ + -Drequire.test.libhadoop + return + fi + + case ${OSTYPE} in + Linux) + # shellcheck disable=SC2086 + echo -Pnative -Drequire.libwebhdfs \ + -Drequire.snappy -Drequire.openssl -Drequire.fuse \ + -Drequire.test.libhadoop + ;; + Darwin) + JANSSON_INCLUDE_DIR=/usr/local/opt/jansson/include + JANSSON_LIBRARY=/usr/local/opt/jansson/lib + export JANSSON_LIBRARY JANSSON_INCLUDE_DIR + # shellcheck disable=SC2086 + echo \ + -Pnative -Drequire.snappy \ + -Drequire.openssl \ + -Dopenssl.prefix=/usr/local/opt/openssl/ \ + -Dopenssl.include=/usr/local/opt/openssl/include \ + -Dopenssl.lib=/usr/local/opt/openssl/lib \ + -Drequire.libwebhdfs -Drequire.test.libhadoop + ;; + *) + # shellcheck disable=SC2086 + echo \ + -Pnative \ + -Drequire.snappy -Drequire.openssl \ + -Drequire.libwebhdfs -Drequire.test.libhadoop + ;; + esac +} + +function personality_modules +{ + local repostatus=$1 + local testtype=$2 + local extra="" + local ordering="normal" + local needflags=false + local flags + local fn + local i + + yetus_debug "Personality: ${repostatus} ${testtype}" + + clear_personality_queue + + case ${testtype} in + asflicense) + # this is very fast and provides the full path if we do it from + # the root of the source + personality_enqueue_module . + return + ;; + checkstyle) + ordering="union" + extra="-DskipTests" + ;; + javac) + ordering="union" + extra="-DskipTests" + needflags=true + + # if something in common changed, we build the whole world + if [[ ${CHANGED_MODULES} =~ hadoop-common ]]; then + yetus_debug "hadoop personality: javac + hadoop-common = ordering set to . " + ordering="." + fi + ;; + javadoc) + if [[ ${repostatus} = patch ]]; then + echo "javadoc pre-reqs:" + for i in hadoop-project \ + hadoop-common-project/hadoop-annotations; do + fn=$(module_file_fragment "${i}") + pushd "${BASEDIR}/${i}" >/dev/null + echo "cd ${i}" + echo_and_redirect "${PATCH_DIR}/maven-${fn}-install.txt" \ + "${MAVEN}" "${MAVEN_ARGS[@]}" install + popd >/dev/null + done + fi + extra="-Pdocs -DskipTests" + ;; + mvninstall) + extra="-DskipTests" + if [[ ${repostatus} = branch ]]; then + ordering=. + fi + ;; + unit) + # As soon as HADOOP-11984 gets committed, + # this code should get uncommented + #if [[ ${TEST_PARALLEL} = "true" ]] ; then + # extra="-Pparallel-tests" + # if [[ -n ${TEST_THREADS:-} ]]; then + # extra="${extra} -DtestsThreadCount=${TEST_THREADS}" + # fi + #fi + needflags=true + hadoop_unittest_prereqs + + verify_needed_test javac + if [[ $? == 0 ]]; then + yetus_debug "hadoop: javac not requested" + verify_needed_test native + if [[ $? == 0 ]]; then + yetus_debug "hadoop: native not requested" + yetus_debug "hadoop: adding -DskipTests to unit test" + extra="-DskipTests" + fi + fi + + verify_needed_test shellcheck + if [[ $? == 0 + && ! ${CHANGED_FILES} =~ \.bats ]]; then + yetus_debug "hadoop: NO shell code change detected; disabling shelltest profile" + extra="${extra} -P!shelltest" + else + extra="${extra} -Pshelltest" + fi + ;; + *) + extra="-DskipTests" + ;; + esac + + if [[ ${needflags} = true ]]; then + flags=$(hadoop_native_flags) + extra="${extra} ${flags}" + fi + + hadoop_module_manipulation ${ordering} + + for module in ${HADOOP_MODULES}; do + # shellcheck disable=SC2086 + personality_enqueue_module ${module} ${extra} + done +} + +function personality_file_tests +{ + local filename=$1 + + yetus_debug "Using Hadoop-specific personality_file_tests" + + if [[ ${filename} =~ src/main/webapp ]]; then + yetus_debug "tests/webapp: ${filename}" + elif [[ ${filename} =~ \.sh + || ${filename} =~ \.cmd + || ${filename} =~ src/scripts + || ${filename} =~ src/test/scripts + ]]; then + yetus_debug "tests/shell: ${filename}" + add_test unit + elif [[ ${filename} =~ \.md$ + || ${filename} =~ \.md\.vm$ + || ${filename} =~ src/site + ]]; then + yetus_debug "tests/site: ${filename}" + add_test site + elif [[ ${filename} =~ \.c$ + || ${filename} =~ \.cc$ + || ${filename} =~ \.h$ + || ${filename} =~ \.hh$ + || ${filename} =~ \.proto$ + || ${filename} =~ \.cmake$ + || ${filename} =~ CMakeLists.txt + ]]; then + yetus_debug "tests/units: ${filename}" + add_test cc + add_test unit + add_test javac + elif [[ ${filename} =~ build.xml$ + || ${filename} =~ pom.xml$ + || ${filename} =~ \.java$ + || ${filename} =~ src/main + ]]; then + yetus_debug "tests/javadoc+units: ${filename}" + add_test javac + add_test javadoc + add_test mvninstall + add_test unit + fi + + if [[ ${filename} =~ src/test ]]; then + yetus_debug "tests" + add_test unit + fi + + if [[ ${filename} =~ \.java$ ]]; then + add_test findbugs + fi +} diff --git a/dev-support/personality/hbase.sh b/dev-support/personality/hbase.sh new file mode 100755 index 0000000000000..ac69226df4506 --- /dev/null +++ b/dev-support/personality/hbase.sh @@ -0,0 +1,240 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#shellcheck disable=SC2034 +PATCH_BRANCH_DEFAULT=master +#shellcheck disable=SC2034 +JIRA_ISSUE_RE='^HBASE-[0-9]+$' +#shellcheck disable=SC2034 +GITHUB_REPO="apache/hbase" +#shellcheck disable=SC2034 +HOW_TO_CONTRIBUTE="" + +# All supported Hadoop versions that we want to test the compilation with +HADOOP2_VERSIONS="2.4.1 2.5.2 2.6.0" + +# Override the maven options +MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}" + +function personality_modules +{ + local repostatus=$1 + local testtype=$2 + local extra="" + + yetus_debug "Personality: ${repostatus} ${testtype}" + + clear_personality_queue + + case ${testtype} in + javac) + personality_enqueue_module . -DskipTests + return + ;; + mvninstall) + extra="-DskipTests -DHBasePatchProcess" + if [[ ${repostatus} == branch ]]; then + personality_enqueue_module . "${extra}" + return + fi + return + ;; + asflicense) + # this is very fast and provides the full path if we do it from + # the root of the source + personality_enqueue_module . -DHBasePatchProcess + return + ;; + unit) + if [[ ${TEST_PARALLEL} == "true" ]] ; then + extra="-Pparallel-tests -DHBasePatchProcess" + if [[ -n ${TEST_THREADS:-} ]]; then + extra="${extra} -DtestsThreadCount=${TEST_THREADS}" + fi + fi + ;; + *) + extra="-DskipTests -DHBasePatchProcess" + ;; + esac + + for module in ${CHANGED_MODULES}; do + + # skip findbugs on hbase-shell + if [[ ${module} == hbase-shell + && ${testtype} == findbugs ]]; then + continue + else + # shellcheck disable=SC2086 + personality_enqueue_module ${module} ${extra} + fi + done +} + +################################################### + +add_plugin hadoopcheck + +function hadoopcheck_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.java$ ]]; then + add_test hadoopcheck + fi +} + +function hadoopcheck_postapply +{ + local HADOOP2_VERSION + local logfile + local count + local result=0 + + big_console_header "Compiling against various Hadoop versions" + + export MAVEN_OPTS="${MAVEN_OPTS}" + for HADOOP2_VERSION in ${HADOOP2_VERSIONS}; do + logfile="${PATCH_DIR}/patch-javac-${HADOOP2_VERSION}.txt" + echo_and_redirect "${logfile}" \ + "${MAVEN}" clean install \ + -DskipTests -DHBasePatchProcess \ + -Dhadoop-two.version="${HADOOP2_VERSION}" + count=$(${GREP} -c ERROR "${logfile}") + if [[ ${count} -gt 0 ]]; then + add_vote_table -1 hadoopcheck "Patch causes ${count} errors with Hadoop v${HADOOP2_VERSION}." + ((result=result+1)) + fi + done + + if [[ ${result} -gt 0 ]]; then + return 1 + fi + + add_vote_table +1 hadoopcheck "Patch does not cause any errors with Hadoop ${HADOOP2_VERSIONS}." + return 0 +} + +###################################### + +add_plugin hbaseprotoc + +function hbaseprotoc_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.proto$ ]]; then + add_test hbaseprotoc + fi +} + +function hbaseprotoc_postapply +{ + local i=0 + local fn + local module + local logfile + local count + local result + + big_console_header "Patch HBase protoc plugin" + + start_clock + + verify_needed_test hbaseprotoc + if [[ $? == 0 ]]; then + echo "Patch does not need hbaseprotoc testing." + return 0 + fi + + personality_modules patch hbaseprotoc + modules_workers patch hbaseprotoc compile -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess + + # shellcheck disable=SC2153 + until [[ $i -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi + module=${MODULE[$i]} + fn=$(module_file_fragment "${module}") + logfile="${PATCH_DIR}/patch-hbaseprotoc-${fn}.txt" + + count=$(${GREP} -c ERROR "${logfile}") + + if [[ ${count} -gt 0 ]]; then + module_status ${i} -1 "patch-hbaseprotoc-${fn}.txt" "Patch generated "\ + "${count} new protoc errors in ${module}." + ((result=result+1)) + fi + ((i=i+1)) + done + + modules_messages patch hbaseprotoc true + if [[ ${result} -gt 0 ]]; then + return 1 + fi + return 0 +} + +###################################### + +add_plugin hbaseanti + +function hbaseanti_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.java$ ]]; then + add_test hbaseanti + fi +} + +function hbaseanti_preapply +{ + local warnings + local result + + big_console_header "Checking for known anti-patterns" + + start_clock + + verify_needed_test hbaseanti + if [[ $? == 0 ]]; then + echo "Patch does not need hbaseanti testing." + return 0 + fi + + warnings=$(${GREP} 'new TreeMap ''' -def clean(str): - return tableclean(re.sub(namePattern, "", str)) +def clean(_str): + return tableclean(re.sub(NAME_PATTERN, "", _str)) -def formatComponents(str): - str = re.sub(namePattern, '', str).replace("'", "") - if str != "": - ret = str - else: - # some markdown parsers don't like empty tables - ret = "." - return clean(ret) +def format_components(_str): + _str = re.sub(NAME_PATTERN, '', _str).replace("'", "") + if _str != "": + ret = _str + else: + # some markdown parsers don't like empty tables + ret = "." + return clean(ret) # convert to utf-8 # protect some known md metachars # or chars that screw up doxia -def tableclean(str): - str=str.encode('utf-8') - str=str.replace("_","\_") - str=str.replace("\r","") - str=str.rstrip() - return str +def tableclean(_str): + _str = _str.encode('utf-8') + _str = _str.replace("_", r"\_") + _str = _str.replace("\r", "") + _str = _str.rstrip() + return _str # same thing as tableclean, # except table metachars are also # escaped as well as more # things we don't want doxia to # screw up -def notableclean(str): - str=tableclean(str) - str=str.replace("|","\|") - str=str.replace("<","\<") - str=str.replace(">","\>") - str=str.replace("*","\*") - str=str.rstrip() - return str +def notableclean(_str): + _str = tableclean(_str) + _str = _str.replace("|", r"\|") + _str = _str.replace("<", r"\<") + _str = _str.replace(">", r"\>") + _str = _str.replace("*", r"\*") + _str = _str.rstrip() + return _str # clean output dir -def cleanOutputDir(dir): - files = os.listdir(dir) +def clean_output_dir(directory): + files = os.listdir(directory) for name in files: - os.remove(os.path.join(dir,name)) - os.rmdir(dir) + os.remove(os.path.join(directory, name)) + os.rmdir(directory) def mstr(obj): - if (obj is None): - return "" - return unicode(obj) - -def buildindex(title,license): - versions=reversed(sorted(glob("[0-9]*.[0-9]*.[0-9]*"))) - with open("index.md","w") as indexfile: - if license is True: - indexfile.write(asflicense) - for v in versions: - indexfile.write("* %s v%s\n" % (title,v)) - for k in ("Changes","Release Notes"): - indexfile.write(" * %s (%s/%s.%s.html)\n" \ - % (k,v,k.upper().replace(" ",""),v)) - indexfile.close() - -class GetVersions: - """ yo """ - def __init__(self,versions, projects): - versions = versions - projects = projects - self.newversions = [] - pp = pprint.PrettyPrinter(indent=4) - at=0 - end=1 - count=100 - versions.sort() - print "Looking for %s through %s"%(versions[0],versions[-1]) - for p in projects: - resp = urllib2.urlopen("https://issues.apache.org/jira/rest/api/2/project/%s/versions"%p) - data = json.loads(resp.read()) - for d in data: - if d['name'][0].isdigit and versions[0] <= d['name'] and d['name'] <= versions[-1]: - print "Adding %s to the list" % d['name'] - self.newversions.append(d['name']) - newlist=list(set(self.newversions)) - self.newversions=newlist - - def getlist(self): - pp = pprint.PrettyPrinter(indent=4) - return(self.newversions) - -class Version: - """Represents a version number""" - def __init__(self, data): - self.mod = False - self.data = data - found = re.match('^((\d+)(\.\d+)*).*$', data) - if (found): - self.parts = [ int(p) for p in found.group(1).split('.') ] - else: - self.parts = [] - # backfill version with zeroes if missing parts - self.parts.extend((0,) * (3 - len(self.parts))) - - def __str__(self): - if (self.mod): - return '.'.join([ str(p) for p in self.parts ]) - return self.data - - def __cmp__(self, other): - return cmp(self.parts, other.parts) - -class Jira: - """A single JIRA""" - - def __init__(self, data, parent): - self.key = data['key'] - self.fields = data['fields'] - self.parent = parent - self.notes = None - self.incompat = None - self.reviewed = None - - def getId(self): - return mstr(self.key) - - def getDescription(self): - return mstr(self.fields['description']) - - def getReleaseNote(self): - if (self.notes is None): - field = self.parent.fieldIdMap['Release Note'] - if (self.fields.has_key(field)): - self.notes=mstr(self.fields[field]) - else: - self.notes=self.getDescription() - return self.notes - - def getPriority(self): - ret = "" - pri = self.fields['priority'] - if(pri is not None): - ret = pri['name'] - return mstr(ret) - - def getAssignee(self): - ret = "" - mid = self.fields['assignee'] - if(mid is not None): - ret = mid['displayName'] - return mstr(ret) - - def getComponents(self): - if (len(self.fields['components'])>0): - return ", ".join([ comp['name'] for comp in self.fields['components'] ]) - else: - return "" - - def getSummary(self): - return self.fields['summary'] - - def getType(self): - ret = "" - mid = self.fields['issuetype'] - if(mid is not None): - ret = mid['name'] - return mstr(ret) - - def getReporter(self): - ret = "" - mid = self.fields['reporter'] - if(mid is not None): - ret = mid['displayName'] - return mstr(ret) - - def getProject(self): - ret = "" - mid = self.fields['project'] - if(mid is not None): - ret = mid['key'] - return mstr(ret) - - def __cmp__(self,other): - selfsplit=self.getId().split('-') - othersplit=other.getId().split('-') - v1=cmp(selfsplit[0],othersplit[0]) - if (v1!=0): - return v1 - else: - if selfsplit[1] < othersplit[1]: + if obj is None: + return "" + return unicode(obj) + +def buildindex(title, asf_license): + versions = reversed(sorted(glob("[0-9]*.[0-9]*.[0-9]*"))) + with open("index.md", "w") as indexfile: + if asf_license is True: + indexfile.write(ASF_LICENSE) + for version in versions: + indexfile.write("* %s v%s\n" % (title, version)) + for k in ("Changes", "Release Notes"): + indexfile.write(" * %s (%s/%s.%s.html)\n" \ + % (k, version, k.upper().replace(" ", ""), version)) + indexfile.close() + +class GetVersions(object): + """ yo """ + def __init__(self, versions, projects): + versions = versions + projects = projects + self.newversions = [] + versions.sort() + print "Looking for %s through %s"%(versions[0], versions[-1]) + for project in projects: + url = "https://issues.apache.org/jira/rest/api/2/project/%s/versions" % project + resp = urllib2.urlopen(url) + datum = json.loads(resp.read()) + for data in datum: + name = data['name'] + if name[0].isdigit and versions[0] <= name and name <= versions[-1]: + print "Adding %s to the list" % name + self.newversions.append(name) + newlist = list(set(self.newversions)) + self.newversions = newlist + + def getlist(self): + return self.newversions + +class Version(object): + """Represents a version number""" + def __init__(self, data): + self.mod = False + self.data = data + found = re.match(r'^((\d+)(\.\d+)*).*$', data) + if found: + self.parts = [int(p) for p in found.group(1).split('.')] + else: + self.parts = [] + # backfill version with zeroes if missing parts + self.parts.extend((0,) * (3 - len(self.parts))) + + def __str__(self): + if self.mod: + return '.'.join([str(p) for p in self.parts]) + return self.data + + def __cmp__(self, other): + return cmp(self.parts, other.parts) + +class Jira(object): + """A single JIRA""" + + def __init__(self, data, parent): + self.key = data['key'] + self.fields = data['fields'] + self.parent = parent + self.notes = None + self.incompat = None + self.reviewed = None + + def get_id(self): + return mstr(self.key) + + def get_description(self): + return mstr(self.fields['description']) + + def get_release_note(self): + if self.notes is None: + field = self.parent.field_id_map['Release Note'] + if self.fields.has_key(field): + self.notes = mstr(self.fields[field]) + else: + self.notes = self.get_description() + return self.notes + + def get_priority(self): + ret = "" + pri = self.fields['priority'] + if pri is not None: + ret = pri['name'] + return mstr(ret) + + def get_assignee(self): + ret = "" + mid = self.fields['assignee'] + if mid is not None: + ret = mid['displayName'] + return mstr(ret) + + def get_components(self): + if len(self.fields['components']) > 0: + return ", ".join([comp['name'] for comp in self.fields['components']]) + else: + return "" + + def get_summary(self): + return self.fields['summary'] + + def get_type(self): + ret = "" + mid = self.fields['issuetype'] + if mid is not None: + ret = mid['name'] + return mstr(ret) + + def get_reporter(self): + ret = "" + mid = self.fields['reporter'] + if mid is not None: + ret = mid['displayName'] + return mstr(ret) + + def get_project(self): + ret = "" + mid = self.fields['project'] + if mid is not None: + ret = mid['key'] + return mstr(ret) + + def __cmp__(self, other): + selfsplit = self.get_id().split('-') + othersplit = other.get_id().split('-') + result = cmp(selfsplit[0], othersplit[0]) + if result != 0: + return result + else: + if selfsplit[1] < othersplit[1]: + return True + elif selfsplit[1] > othersplit[1]: + return False + return False + + def get_incompatible_change(self): + if self.incompat is None: + field = self.parent.field_id_map['Hadoop Flags'] + self.reviewed = False + self.incompat = False + if self.fields.has_key(field): + if self.fields[field]: + for flag in self.fields[field]: + if flag['value'] == "Incompatible change": + self.incompat = True + if flag['value'] == "Reviewed": + self.reviewed = True + return self.incompat + + def check_missing_component(self): + if len(self.fields['components']) > 0: + return False return True - elif selfsplit[1] > othersplit[1]: + + def check_missing_assignee(self): + if self.fields['assignee'] is not None: + return False + return True + + def check_version_string(self): + field = self.parent.field_id_map['Fix Version/s'] + for ver in self.fields[field]: + found = re.match(r'^((\d+)(\.\d+)*).*$|^(\w+\-\d+)$', ver['name']) + if not found: + return True return False - return False - - def getIncompatibleChange(self): - if (self.incompat is None): - field = self.parent.fieldIdMap['Hadoop Flags'] - self.reviewed=False - self.incompat=False - if (self.fields.has_key(field)): - if self.fields[field]: - for hf in self.fields[field]: - if hf['value'] == "Incompatible change": - self.incompat=True - if hf['value'] == "Reviewed": - self.reviewed=True - return self.incompat - - def checkMissingComponent(self): - if (len(self.fields['components'])>0): - return False - return True - - def checkMissingAssignee(self): - if (self.fields['assignee'] is not None): - return False - return True - - def checkVersionString(self): - field = self.parent.fieldIdMap['Fix Version/s'] - for h in self.fields[field]: - found = re.match('^((\d+)(\.\d+)*).*$|^(\w+\-\d+)$', h['name']) - if not found: - return True - return False - - def getReleaseDate(self,version): - for j in range(len(self.fields['fixVersions'])): - if self.fields['fixVersions'][j]==version: - return(self.fields['fixVersions'][j]['releaseDate']) - return None - -class JiraIter: - """An Iterator of JIRAs""" - - def __init__(self, version, projects): - self.version = version - self.projects = projects - v=str(version).replace("-SNAPSHOT","") - - resp = urllib2.urlopen("https://issues.apache.org/jira/rest/api/2/field") - data = json.loads(resp.read()) - - self.fieldIdMap = {} - for part in data: - self.fieldIdMap[part['name']] = part['id'] - - self.jiras = [] - at=0 - end=1 - count=100 - while (at < end): - params = urllib.urlencode({'jql': "project in ('"+"' , '".join(projects)+"') and fixVersion in ('"+v+"') and resolution = Fixed", 'startAt':at, 'maxResults':count}) - resp = urllib2.urlopen("https://issues.apache.org/jira/rest/api/2/search?%s"%params) - data = json.loads(resp.read()) - if (data.has_key('errorMessages')): - raise Exception(data['errorMessages']) - at = data['startAt'] + data['maxResults'] - end = data['total'] - self.jiras.extend(data['issues']) - - needaversion=False - if v not in releaseVersion: - needaversion=True - - if needaversion is True: - for i in range(len(data['issues'])): - for j in range(len(data['issues'][i]['fields']['fixVersions'])): - if 'releaseDate' in data['issues'][i]['fields']['fixVersions'][j]: - releaseVersion[data['issues'][i]['fields']['fixVersions'][j]['name']]=\ - data['issues'][i]['fields']['fixVersions'][j]['releaseDate'] - - self.iter = self.jiras.__iter__() - - def __iter__(self): - return self - - def next(self): - data = self.iter.next() - j = Jira(data, self) - return j - -class Outputs: - """Several different files to output to at the same time""" - - def __init__(self, base_file_name, file_name_pattern, keys, params={}): - self.params = params - self.base = open(base_file_name%params, 'w') - self.others = {} - for key in keys: - both = dict(params) - both['key'] = key - self.others[key] = open(file_name_pattern%both, 'w') - - def writeAll(self, pattern): - both = dict(self.params) - both['key'] = '' - self.base.write(pattern%both) - for key in self.others.keys(): - both = dict(self.params) - both['key'] = key - self.others[key].write(pattern%both) - - def writeKeyRaw(self, key, str): - self.base.write(str) - if (self.others.has_key(key)): - self.others[key].write(str) - - def close(self): - self.base.close() - for fd in self.others.values(): - fd.close() - - def writeList(self, mylist): - for jira in sorted(mylist): - line = '| [%s](https://issues.apache.org/jira/browse/%s) | %s | %s | %s | %s | %s |\n' \ - % (notableclean(jira.getId()), notableclean(jira.getId()), - notableclean(jira.getSummary()), - notableclean(jira.getPriority()), - formatComponents(jira.getComponents()), - notableclean(jira.getReporter()), - notableclean(jira.getAssignee())) - self.writeKeyRaw(jira.getProject(), line) + + def get_release_date(self, version): + fix_versions = self.fields['fixVersions'] + for j in range(len(fix_versions)): + if fix_versions[j] == version: + return fix_versions[j]['releaseDate'] + return None + +class JiraIter(object): + """An Iterator of JIRAs""" + + def __init__(self, version, projects): + self.version = version + self.projects = projects + ver = str(version).replace("-SNAPSHOT", "") + + resp = urllib2.urlopen("https://issues.apache.org/jira/rest/api/2/field") + data = json.loads(resp.read()) + + self.field_id_map = {} + for part in data: + self.field_id_map[part['name']] = part['id'] + + self.jiras = [] + pos = 0 + end = 1 + count = 100 + while pos < end: + pjs = "','".join(projects) + jql = "project in ('%s') and fixVersion in ('%s') and resolution = Fixed" % (pjs, ver) + params = urllib.urlencode({'jql': jql, 'startAt':pos, 'maxResults':count}) + resp = urllib2.urlopen("https://issues.apache.org/jira/rest/api/2/search?%s" % params) + data = json.loads(resp.read()) + if data.has_key('error_messages'): + raise Exception(data['error_messages']) + pos = data['startAt'] + data['maxResults'] + end = data['total'] + self.jiras.extend(data['issues']) + + needaversion = False + if ver not in RELEASE_VERSION: + needaversion = True + + if needaversion is True: + issues = data['issues'] + for i in range(len(issues)): + fix_versions = issues[i]['fields']['fixVersions'] + for j in range(len(fix_versions)): + fields = fix_versions[j] + if 'releaseDate' in fields: + RELEASE_VERSION[fields['name']] = fields['releaseDate'] + + self.iter = self.jiras.__iter__() + + def __iter__(self): + return self + + def next(self): + data = self.iter.next() + j = Jira(data, self) + return j + +class Outputs(object): + """Several different files to output to at the same time""" + + def __init__(self, base_file_name, file_name_pattern, keys, params=None): + if params is None: + params = {} + self.params = params + self.base = open(base_file_name%params, 'w') + self.others = {} + for key in keys: + both = dict(params) + both['key'] = key + self.others[key] = open(file_name_pattern%both, 'w') + + def write_all(self, pattern): + both = dict(self.params) + both['key'] = '' + self.base.write(pattern%both) + for key in self.others.keys(): + both = dict(self.params) + both['key'] = key + self.others[key].write(pattern%both) + + def write_key_raw(self, key, _str): + self.base.write(_str) + if self.others.has_key(key): + self.others[key].write(_str) + + def close(self): + self.base.close() + for value in self.others.values(): + value.close() + + def write_list(self, mylist): + for jira in sorted(mylist): + line = '| [%s](https://issues.apache.org/jira/browse/%s) | %s | %s | %s | %s | %s |\n' + line = line % (notableclean(jira.get_id()), + notableclean(jira.get_id()), + notableclean(jira.get_summary()), + notableclean(jira.get_priority()), + format_components(jira.get_components()), + notableclean(jira.get_reporter()), + notableclean(jira.get_assignee())) + self.write_key_raw(jira.get_project(), line) def main(): - parser = OptionParser(usage="usage: %prog --project PROJECT [--project PROJECT] --version VERSION [--version VERSION2 ...]", - epilog= - "Markdown-formatted CHANGES and RELEASENOTES files will be stored in a directory" - " named after the highest version provided.") - parser.add_option("-i","--index", dest="index", action="store_true", - default=False, help="build an index file") - parser.add_option("-l","--license", dest="license", action="store_false", - default=True, help="Add an ASF license") - parser.add_option("-n","--lint", dest="lint", action="store_true", - help="use lint flag to exit on failures") - parser.add_option("-p", "--project", dest="projects", - action="append", type="string", - help="projects in JIRA to include in releasenotes", metavar="PROJECT") - parser.add_option("-r", "--range", dest="range", action="store_true", - default=False, help="Given versions are a range") - parser.add_option("-t", "--projecttitle", dest="title", - type="string", - help="Title to use for the project (default is Apache PROJECT)") - parser.add_option("-u","--usetoday", dest="usetoday", action="store_true", - default=False, help="use current date for unreleased versions") - parser.add_option("-v", "--version", dest="versions", - action="append", type="string", - help="versions in JIRA to include in releasenotes", metavar="VERSION") - (options, args) = parser.parse_args() - - if (options.versions is None): - options.versions = [] - - if (len(args) > 2): - options.versions.append(args[2]) - - if (len(options.versions) <= 0): - parser.error("At least one version needs to be supplied") - - proxy = urllib2.ProxyHandler() - opener = urllib2.build_opener(proxy) - urllib2.install_opener(opener) - - projects = options.projects - - if (options.range is True): - versions = [ Version(v) for v in GetVersions(options.versions, projects).getlist() ] - else: - versions = [ Version(v) for v in options.versions ] - versions.sort(); - - if (options.title is None): - title=projects[0] - else: - title=options.title - - haderrors=False - - for v in versions: - vstr=str(v) - jlist = JiraIter(vstr,projects) - - if vstr in releaseVersion: - reldate=releaseVersion[vstr] - elif options.usetoday: - reldate=strftime("%Y-%m-%d", gmtime()) + usage = "usage: %prog --project PROJECT [--project PROJECT] --version VERSION [--version VERSION2 ...]" + parser = OptionParser(usage=usage, + epilog="Markdown-formatted CHANGES and RELEASENOTES files will be stored" + "in a directory named after the highest version provided.") + parser.add_option("-i", "--index", dest="index", action="store_true", + default=False, help="build an index file") + parser.add_option("-l", "--license", dest="license", action="store_false", + default=True, help="Add an ASF license") + parser.add_option("-n", "--lint", dest="lint", action="store_true", + help="use lint flag to exit on failures") + parser.add_option("-p", "--project", dest="projects", + action="append", type="string", + help="projects in JIRA to include in releasenotes", metavar="PROJECT") + parser.add_option("-r", "--range", dest="range", action="store_true", + default=False, help="Given versions are a range") + parser.add_option("-t", "--projecttitle", dest="title", type="string", + help="Title to use for the project (default is Apache PROJECT)") + parser.add_option("-u", "--usetoday", dest="usetoday", action="store_true", + default=False, help="use current date for unreleased versions") + parser.add_option("-v", "--version", dest="versions", action="append", type="string", + help="versions in JIRA to include in releasenotes", metavar="VERSION") + (options, _) = parser.parse_args() + + if options.versions is None: + parser.error("At least one version needs to be supplied") + + proxy = urllib2.ProxyHandler() + opener = urllib2.build_opener(proxy) + urllib2.install_opener(opener) + + projects = options.projects + if projects is None: + parser.error("At least one project needs to be supplied") + + if options.range is True: + versions = [Version(v) for v in GetVersions(options.versions, projects).getlist()] + else: + versions = [Version(v) for v in options.versions] + versions.sort() + + if options.title is None: + title = projects[0] else: - reldate="Unreleased" - - if not os.path.exists(vstr): - os.mkdir(vstr) - - reloutputs = Outputs("%(ver)s/RELEASENOTES.%(ver)s.md", - "%(ver)s/RELEASENOTES.%(key)s.%(ver)s.md", - [], {"ver":v, "date":reldate, "title":title}) - choutputs = Outputs("%(ver)s/CHANGES.%(ver)s.md", - "%(ver)s/CHANGES.%(key)s.%(ver)s.md", - [], {"ver":v, "date":reldate, "title":title}) - - if (options.license is True): - reloutputs.writeAll(asflicense) - choutputs.writeAll(asflicense) - - relhead = '# %(title)s %(key)s %(ver)s Release Notes\n\n' \ - 'These release notes cover new developer and user-facing incompatibilities, features, and major improvements.\n\n' - chhead = '# %(title)s Changelog\n\n' \ - '## Release %(ver)s - %(date)s\n'\ - '\n' - - reloutputs.writeAll(relhead) - choutputs.writeAll(chhead) - errorCount=0 - warningCount=0 - lintMessage="" - incompatlist=[] - buglist=[] - improvementlist=[] - newfeaturelist=[] - subtasklist=[] - tasklist=[] - testlist=[] - otherlist=[] - - for jira in sorted(jlist): - if jira.getIncompatibleChange(): - incompatlist.append(jira) - elif jira.getType() == "Bug": - buglist.append(jira) - elif jira.getType() == "Improvement": - improvementlist.append(jira) - elif jira.getType() == "New Feature": - newfeaturelist.append(jira) - elif jira.getType() == "Sub-task": - subtasklist.append(jira) - elif jira.getType() == "Task": - tasklist.append(jira) - elif jira.getType() == "Test": - testlist.append(jira) - else: - otherlist.append(jira) - - line = '* [%s](https://issues.apache.org/jira/browse/%s) | *%s* | **%s**\n' \ - % (notableclean(jira.getId()), notableclean(jira.getId()), notableclean(jira.getPriority()), - notableclean(jira.getSummary())) - - if (jira.getIncompatibleChange()) and (len(jira.getReleaseNote())==0): - warningCount+=1 - reloutputs.writeKeyRaw(jira.getProject(),"\n---\n\n") - reloutputs.writeKeyRaw(jira.getProject(), line) - line ='\n**WARNING: No release note provided for this incompatible change.**\n\n' - lintMessage += "\nWARNING: incompatible change %s lacks release notes." % (notableclean(jira.getId())) - reloutputs.writeKeyRaw(jira.getProject(), line) - - if jira.checkVersionString(): - warningCount+=1 - lintMessage += "\nWARNING: Version string problem for %s " % jira.getId() - - if (jira.checkMissingComponent() or jira.checkMissingAssignee()): - errorCount+=1 - errorMessage=[] - jira.checkMissingComponent() and errorMessage.append("component") - jira.checkMissingAssignee() and errorMessage.append("assignee") - lintMessage += "\nERROR: missing %s for %s " % (" and ".join(errorMessage) , jira.getId()) - - if (len(jira.getReleaseNote())>0): - reloutputs.writeKeyRaw(jira.getProject(),"\n---\n\n") - reloutputs.writeKeyRaw(jira.getProject(), line) - line ='\n%s\n\n' % (tableclean(jira.getReleaseNote())) - reloutputs.writeKeyRaw(jira.getProject(), line) - - if (options.lint is True): - print lintMessage - print "=======================================" - print "%s: Error:%d, Warning:%d \n" % (vstr, errorCount, warningCount) - if (errorCount>0): - haderrors=True - cleanOutputDir(vstr) - continue - - reloutputs.writeAll("\n\n") - reloutputs.close() - - choutputs.writeAll("### INCOMPATIBLE CHANGES:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(incompatlist) - - choutputs.writeAll("\n\n### NEW FEATURES:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(newfeaturelist) - - choutputs.writeAll("\n\n### IMPROVEMENTS:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(improvementlist) - - choutputs.writeAll("\n\n### BUG FIXES:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(buglist) - - choutputs.writeAll("\n\n### TESTS:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(testlist) - - choutputs.writeAll("\n\n### SUB-TASKS:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(subtasklist) - - choutputs.writeAll("\n\n### OTHER:\n\n") - choutputs.writeAll("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") - choutputs.writeAll("|:---- |:---- | :--- |:---- |:---- |:---- |\n") - choutputs.writeList(otherlist) - choutputs.writeList(tasklist) - - choutputs.writeAll("\n\n") - choutputs.close() - - if options.index: - buildindex(title,options.license) - - if haderrors is True: - sys.exit(1) + title = options.title + + haderrors = False + + for version in versions: + vstr = str(version) + jlist = JiraIter(vstr, projects) + + if vstr in RELEASE_VERSION: + reldate = RELEASE_VERSION[vstr] + elif options.usetoday: + reldate = strftime("%Y-%m-%d", gmtime()) + else: + reldate = "Unreleased (as of %s)" % strftime("%Y-%m-%d", gmtime()) + + if not os.path.exists(vstr): + os.mkdir(vstr) + + reloutputs = Outputs("%(ver)s/RELEASENOTES.%(ver)s.md", + "%(ver)s/RELEASENOTES.%(key)s.%(ver)s.md", + [], {"ver":version, "date":reldate, "title":title}) + choutputs = Outputs("%(ver)s/CHANGES.%(ver)s.md", + "%(ver)s/CHANGES.%(key)s.%(ver)s.md", + [], {"ver":version, "date":reldate, "title":title}) + + if options.license is True: + reloutputs.write_all(ASF_LICENSE) + choutputs.write_all(ASF_LICENSE) + + relhead = '# %(title)s %(key)s %(ver)s Release Notes\n\n' \ + 'These release notes cover new developer and user-facing ' \ + 'incompatibilities, features, and major improvements.\n\n' + chhead = '# %(title)s Changelog\n\n' \ + '## Release %(ver)s - %(date)s\n'\ + '\n' + + reloutputs.write_all(relhead) + choutputs.write_all(chhead) + error_count = 0 + warning_count = 0 + lint_message = "" + incompatlist = [] + buglist = [] + improvementlist = [] + newfeaturelist = [] + subtasklist = [] + tasklist = [] + testlist = [] + otherlist = [] + + for jira in sorted(jlist): + if jira.get_incompatible_change(): + incompatlist.append(jira) + elif jira.get_type() == "Bug": + buglist.append(jira) + elif jira.get_type() == "Improvement": + improvementlist.append(jira) + elif jira.get_type() == "New Feature": + newfeaturelist.append(jira) + elif jira.get_type() == "Sub-task": + subtasklist.append(jira) + elif jira.get_type() == "Task": + tasklist.append(jira) + elif jira.get_type() == "Test": + testlist.append(jira) + else: + otherlist.append(jira) + + line = '* [%s](https://issues.apache.org/jira/browse/%s) | *%s* | **%s**\n' \ + % (notableclean(jira.get_id()), notableclean(jira.get_id()), + notableclean(jira.get_priority()), notableclean(jira.get_summary())) + + if jira.get_incompatible_change() and len(jira.get_release_note()) == 0: + warning_count += 1 + reloutputs.write_key_raw(jira.get_project(), "\n---\n\n") + reloutputs.write_key_raw(jira.get_project(), line) + line = '\n**WARNING: No release note provided for this incompatible change.**\n\n' + lint_message += "\nWARNING: incompatible change %s lacks release notes." % \ + (notableclean(jira.get_id())) + reloutputs.write_key_raw(jira.get_project(), line) + + if jira.check_version_string(): + warning_count += 1 + lint_message += "\nWARNING: Version string problem for %s " % jira.get_id() + + if jira.check_missing_component() or jira.check_missing_assignee(): + error_count += 1 + error_message = [] + if jira.check_missing_component(): + error_message.append("component") + if jira.check_missing_assignee(): + error_message.append("assignee") + lint_message += "\nERROR: missing %s for %s " \ + % (" and ".join(error_message), jira.get_id()) + + if len(jira.get_release_note()) > 0: + reloutputs.write_key_raw(jira.get_project(), "\n---\n\n") + reloutputs.write_key_raw(jira.get_project(), line) + line = '\n%s\n\n' % (tableclean(jira.get_release_note())) + reloutputs.write_key_raw(jira.get_project(), line) + + if options.lint is True: + print lint_message + print "=======================================" + print "%s: Error:%d, Warning:%d \n" % (vstr, error_count, warning_count) + if error_count > 0: + haderrors = True + clean_output_dir(vstr) + continue + + reloutputs.write_all("\n\n") + reloutputs.close() + + choutputs.write_all("### INCOMPATIBLE CHANGES:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(incompatlist) + + choutputs.write_all("\n\n### NEW FEATURES:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(newfeaturelist) + + choutputs.write_all("\n\n### IMPROVEMENTS:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(improvementlist) + + choutputs.write_all("\n\n### BUG FIXES:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(buglist) + + choutputs.write_all("\n\n### TESTS:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(testlist) + + choutputs.write_all("\n\n### SUB-TASKS:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(subtasklist) + + choutputs.write_all("\n\n### OTHER:\n\n") + choutputs.write_all("| JIRA | Summary | Priority | Component | Reporter | Contributor |\n") + choutputs.write_all("|:---- |:---- | :--- |:---- |:---- |:---- |\n") + choutputs.write_list(otherlist) + choutputs.write_list(tasklist) + + choutputs.write_all("\n\n") + choutputs.close() + + if options.index: + buildindex(title, options.license) + + if haderrors is True: + sys.exit(1) if __name__ == "__main__": - main() + main() diff --git a/dev-support/smart-apply-patch.sh b/dev-support/smart-apply-patch.sh index ebcb660c511b8..e7ac9eb19f8cf 100755 --- a/dev-support/smart-apply-patch.sh +++ b/dev-support/smart-apply-patch.sh @@ -11,177 +11,537 @@ # See the License for the specific language governing permissions and # limitations under the License. -# -# Determine if the git diff patch file has prefixes. -# These files are generated via "git diff" *without* the --no-prefix option. -# -# We can apply these patches more easily because we know that the a/ and b/ -# prefixes in the "diff" lines stands for the project root directory. -# So we don't have to hunt for the project root. -# And of course, we know that the patch file was generated using git, so we -# know git apply can handle it properly. -# -# Arguments: git diff file name. -# Return: 0 if it is a git diff with prefix; 1 otherwise. -# -has_prefix() { - awk '/^diff --git / { if ($3 !~ "^a/" || $4 !~ "^b/") { exit 1 } } - /^\+{3}|-{3} / { if ($2 !~ "^[ab]/" && $2 !~ "^/dev/null") { exit 1 } }' "$1" - return $? -} +# Make sure that bash version meets the pre-requisite -PATCH_FILE=$1 -DRY_RUN=$2 -if [ -z "$PATCH_FILE" ]; then - echo usage: $0 patch-file +if [[ -z "${BASH_VERSINFO}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." exit 1 fi -TMPDIR=${TMPDIR:-/tmp} -PATCH=${PATCH:-patch} # allow overriding patch binary +RESULT=0 -# Cleanup handler for temporary files -TOCLEAN="" -cleanup() { - rm $TOCLEAN - exit $1 +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_error +{ + echo "$*" 1>&2 } -trap "cleanup 1" HUP INT QUIT TERM -# Allow passing "-" for stdin patches -if [ "$PATCH_FILE" == "-" ]; then - PATCH_FILE="$TMPDIR/smart-apply.in.$RANDOM" - cat /dev/fd/0 > $PATCH_FILE - TOCLEAN="$TOCLEAN $PATCH_FILE" -fi +## @description Print a message to stderr if --debug is turned on +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_debug +{ + if [[ -n "${YETUS_SHELL_SCRIPT_DEBUG}" ]]; then + echo "[$(date) DEBUG]: $*" 1>&2 + fi +} -ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$' -if [[ ${PATCH_FILE} =~ ^http || ${PATCH_FILE} =~ ${ISSUE_RE} ]]; then - # Allow downloading of patches - PFILE="$TMPDIR/smart-apply.in.$RANDOM" - TOCLEAN="$TOCLEAN $PFILE" - if [[ ${PATCH_FILE} =~ ^http ]]; then - patchURL="${PATCH_FILE}" - else # Get URL of patch from JIRA - wget -q -O "${PFILE}" "http://issues.apache.org/jira/browse/${PATCH_FILE}" - if [[ $? != 0 ]]; then - echo "Unable to determine what ${PATCH_FILE} may reference." 1>&2 - cleanup 1 - elif [[ $(grep -c 'Patch Available' "${PFILE}") == 0 ]]; then - echo "${PATCH_FILE} is not \"Patch Available\". Exiting." 1>&2 - cleanup 1 - fi - relativePatchURL=$(grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' "${PFILE}" | grep -v -e 'htm[l]*$' | sort | tail -1 | grep -o '/jira/secure/attachment/[0-9]*/[^"]*') - patchURL="http://issues.apache.org${relativePatchURL}" +## @description Clean the filesystem as appropriate and then exit +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +function cleanup_and_exit +{ + local result=$1 + + if [[ ${PATCH_DIR} =~ ^/tmp/apply-patch + && -d ${PATCH_DIR} ]]; then + rm -rf "${PATCH_DIR}" + fi + + # shellcheck disable=SC2086 + exit ${result} +} + +## @description Setup the default global variables +## @audience public +## @stability stable +## @replaceable no +function setup_defaults +{ + PATCHURL="" + OSTYPE=$(uname -s) + + # Solaris needs POSIX, not SVID + case ${OSTYPE} in + SunOS) + AWK=${AWK:-/usr/xpg4/bin/awk} + SED=${SED:-/usr/xpg4/bin/sed} + CURL=${CURL:-curl} + GIT=${GIT:-git} + GREP=${GREP:-/usr/xpg4/bin/grep} + PATCH=${PATCH:-/usr/gnu/bin/patch} + DIFF=${DIFF:-/usr/gnu/bin/diff} + FILE=${FILE:-file} + ;; + *) + AWK=${AWK:-awk} + SED=${SED:-sed} + CURL=${CURL:-curl} + GIT=${GIT:-git} + GREP=${GREP:-grep} + PATCH=${PATCH:-patch} + DIFF=${DIFF:-diff} + FILE=${FILE:-file} + ;; + esac + + DRYRUNMODE=false + PATCH_DIR=/tmp + while [[ -e ${PATCH_DIR} ]]; do + PATCH_DIR=/tmp/apply-patch-${RANDOM}.${RANDOM} + done + PATCHMODES=("git" "patch") + PATCHMODE="" + PATCHPREFIX=0 +} + +## @description Print the usage information +## @audience public +## @stability stable +## @replaceable no +function yetus_usage +{ + echo "Usage: apply-patch.sh [options] patch-file | issue-number | http" + echo + echo "--debug If set, then output some extra stuff to stderr" + echo "--dry-run Check for patch viability without applying" + echo "--patch-dir= The directory for working and output files (default '/tmp/apply-patch-(random))" + echo + echo "Shell binary overrides:" + echo "--file-cmd= The 'file' command to use (default 'file')" + echo "--grep-cmd= The 'grep' command to use (default 'grep')" + echo "--git-cmd= The 'git' command to use (default 'git')" + echo "--patch-cmd= The GNU-compatible 'patch' command to use (default 'patch')" + echo "--curl-cmd= The 'curl' command to use (default 'curl')" +} + +## @description Interpret the command line parameters +## @audience private +## @stability stable +## @replaceable no +## @params $@ +## @return May exit on failure +function parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --debug) + YETUS_SHELL_SCRIPT_DEBUG=true + ;; + --dry-run) + DRYRUNMODE=true + ;; + --file-cmd=*) + FILE=${i#*=} + ;; + --git-cmd=*) + GIT=${i#*=} + ;; + --grep-cmd=*) + GREP=${i#*=} + ;; + --help|-help|-h|help|--h|--\?|-\?|\?) + yetus_usage + exit 0 + ;; + --patch-cmd=*) + PATCH=${i#*=} + ;; + --patch-dir=*) + PATCH_DIR=${i#*=} + ;; + --curl-cmd=*) + CURL=${i#*=} + ;; + --*) + ## PATCH_OR_ISSUE can't be a --. So this is probably + ## a plugin thing. + continue + ;; + *) + PATCH_OR_ISSUE=${i#*=} + ;; + esac + done + + if [[ ! -d ${PATCH_DIR} ]]; then + mkdir -p "${PATCH_DIR}" + if [[ $? != 0 ]] ; then + yetus_error "ERROR: Unable to create ${PATCH_DIR}" + cleanup_and_exit 1 + fi fi - if [[ -n $DRY_RUN ]]; then - echo "Downloading ${patchURL}" +} + +## @description Given a possible patch file, guess if it's a patch file without using smart-apply-patch +## @audience private +## @stability evolving +## @param path to patch file to test +## @return 0 we think it's a patch file +## @return 1 we think it's not a patch file +function guess_patch_file +{ + local patch=$1 + local fileOutput + + yetus_debug "Trying to guess is ${patch} is a patch file." + fileOutput=$("${FILE}" "${patch}") + if [[ $fileOutput =~ \ diff\ ]]; then + yetus_debug "file magic says it's a diff." + return 0 fi - wget -q -O "${PFILE}" "${patchURL}" - if [[ $? != 0 ]]; then - echo "${PATCH_FILE} could not be downloaded." 1>&2 - cleanup 1 + fileOutput=$(head -n 1 "${patch}" | "${GREP}" -E "^(From [a-z0-9]* Mon Sep 17 00:00:00 2001)|(diff .*)|(Index: .*)$") + if [[ $? == 0 ]]; then + yetus_debug "first line looks like a patch file." + return 0 fi - PATCH_FILE="${PFILE}" -fi + return 1 +} -# Case for git-diff patches -if grep -q "^diff --git" "${PATCH_FILE}"; then - GIT_FLAGS="--binary -v" - if has_prefix "$PATCH_FILE"; then - GIT_FLAGS="$GIT_FLAGS -p1" +## @description Given ${PATCH_ISSUE}, determine what type of patch file is in use, and do the +## @description necessary work to place it into ${PATCH_DIR}/patch. +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure, may exit +function locate_patch +{ + local notSureIfPatch=false + yetus_debug "locate patch" + + # Allow passing "-" for stdin patches + if [[ ${PATCH_OR_ISSUE} == - ]]; then + PATCH_FILE="${PATCH_DIR}/patch" + cat /dev/fd/0 > "${PATCH_FILE}" + elif [[ -f ${PATCH_OR_ISSUE} ]]; then + PATCH_FILE="${PATCH_OR_ISSUE}" else - GIT_FLAGS="$GIT_FLAGS -p0" + if [[ ${PATCH_OR_ISSUE} =~ ^http ]]; then + echo "Patch is being downloaded at $(date) from" + PATCHURL="${PATCH_OR_ISSUE}" + else + ${CURL} --silent \ + --output "${PATCH_DIR}/jira" \ + --location \ + "http://issues.apache.org/jira/browse/${PATCH_OR_ISSUE}" + case $? in + 0) + ;; + 2) + yetus_error "ERROR: .curlrc/.netrc parsing error." + cleanup_and_exit 1 + ;; + 3) + yetus_error "ERROR: File IO error." + cleanup_and_exit 1 + ;; + 4) + yetus_error "ERROR: URL ${PATCH_OR_ISSUE} is unreachable." + cleanup_and_exit 1 + ;; + *) + yetus_error "ERROR: Unable to fetch ${PATCH_OR_ISSUE}." + cleanup_and_exit 1 + ;; + esac + + if [[ -z "${PATCH_FILE}" ]]; then + if [[ $(${GREP} -c 'Patch Available' "${PATCH_DIR}/jira") == 0 ]] ; then + if [[ ${JENKINS} == true ]]; then + yetus_error "ERROR: ${PATCH_OR_ISSUE} is not \"Patch Available\"." + cleanup_and_exit 1 + else + yetus_error "WARNING: ${PATCH_OR_ISSUE} is not \"Patch Available\"." + fi + fi + + #shellcheck disable=SC2016 + relativePatchURL=$(${AWK} 'match($0,"\"/jira/secure/attachment/[0-9]*/[^\"]*"){print substr($0,RSTART+1,RLENGTH-1)}' "${PATCH_DIR}/jira" | + ${GREP} -v -e 'htm[l]*$' | sort | tail -1) + PATCHURL="http://issues.apache.org${relativePatchURL}" + if [[ ! ${PATCHURL} =~ \.patch$ ]]; then + notSureIfPatch=true + fi + echo "${ISSUE} patch is being downloaded at $(date) from" + fi + fi + if [[ -z "${PATCH_FILE}" ]]; then + ${CURL} -q --location --output "${PATCH_DIR}/patch" "${PATCHURL}" + if [[ $? != 0 ]];then + yetus_error "ERROR: ${PATCH_OR_ISSUE} could not be downloaded." + cleanup_and_exit 1 + fi + PATCH_FILE="${PATCH_DIR}/patch" + fi fi - if [[ -z $DRY_RUN ]]; then - GIT_FLAGS="$GIT_FLAGS --stat --apply" - echo Going to apply git patch with: git apply "${GIT_FLAGS}" - else - GIT_FLAGS="$GIT_FLAGS --check" + + if [[ ! -f "${PATCH_DIR}/patch" ]]; then + cp "${PATCH_FILE}" "${PATCH_DIR}/patch" + if [[ $? == 0 ]] ; then + echo "Patch file ${PATCH_FILE} copied to ${PATCH_DIR}" + else + yetus_error "ERROR: Could not copy ${PATCH_FILE} to ${PATCH_DIR}" + cleanup_and_exit 1 + fi fi - # shellcheck disable=SC2086 - git apply ${GIT_FLAGS} "${PATCH_FILE}" - if [[ $? == 0 ]]; then - cleanup 0 + + if [[ ! -f "${PATCH_DIR}/patch" ]]; then + cp "${PATCH_FILE}" "${PATCH_DIR}/patch" + if [[ $? == 0 ]] ; then + echo "Patch file ${PATCH_FILE} copied to ${PATCH_DIR}" + else + yetus_error "ERROR: Could not copy ${PATCH_FILE} to ${PATCH_DIR}" + cleanup_and_exit 1 + fi fi - echo "git apply failed. Going to apply the patch with: ${PATCH}" -fi -# Come up with a list of changed files into $TMP -TMP="$TMPDIR/smart-apply.paths.$RANDOM" -TOCLEAN="$TOCLEAN $TMP" + if [[ ${notSureIfPatch} == "true" ]]; then + guess_patch_file "${PATCH_FILE}" + if [[ $? != 0 ]]; then + yetus_error "ERROR: ${PATCHURL} is not a patch file." + cleanup_and_exit 1 + else + yetus_debug "The patch ${PATCHURL} was not named properly, but it looks like a patch file. proceeding, but issue/branch matching might go awry." + fi + fi +} + +## @description if patch-level zero, then verify we aren't +## @description just adding files +## @audience public +## @stability stable +## @param filename +## @param command +## @param [..] +## @replaceable no +## @returns $? +function verify_zero +{ + local logfile=$1 + shift + local dir + + # don't return /dev/null + # shellcheck disable=SC2016 + changed_files1=$(${AWK} 'function p(s){if(s!~"^/dev/null"){print s}} + /^diff --git / { p($3); p($4) } + /^(\+\+\+|---) / { p($2) }' "${PATCH_DIR}/patch" | sort -u) + + # maybe we interpreted the patch wrong? check the log file + # shellcheck disable=SC2016 + changed_files2=$(${GREP} -E '^[cC]heck' "${logfile}" \ + | ${AWK} '{print $3}' \ + | ${SED} -e 's,\.\.\.$,,g') -if $PATCH -p0 -E --dry-run < $PATCH_FILE 2>&1 > $TMP; then - PLEVEL=0 - #if the patch applied at P0 there is the possability that all we are doing - # is adding new files and they would apply anywhere. So try to guess the - # correct place to put those files. + for filename in ${changed_files1} ${changed_files2}; do - TMP2="$TMPDIR/smart-apply.paths.2.$RANDOM" - TOCLEAN="$TOCLEAN $TMP2" + # leading prefix = bad + if [[ ${filename} =~ ^(a|b)/ ]]; then + return 1 + fi + + # touching an existing file is proof enough + # that pl=0 is good + if [[ -f ${filename} ]]; then + return 0 + fi - egrep '^patching file |^checking file ' $TMP | awk '{print $3}' | grep -v /dev/null | sort -u > $TMP2 + dir=$(dirname "${filename}" 2>/dev/null) + if [[ -n ${dir} && -d ${dir} ]]; then + return 0 + fi + done - if [ ! -s $TMP2 ]; then - echo "Error: Patch dryrun couldn't detect changes the patch would make. Exiting." - cleanup 1 + # ¯\_(ツ)_/¯ - no way for us to know, all new files with no prefix! + yetus_error "WARNING: Patch only adds files; using patch level ${PATCHPREFIX}" + return 0 +} + +## @description run the command, sending stdout and stderr to the given filename +## @audience public +## @stability stable +## @param filename +## @param command +## @param [..] +## @replaceable no +## @returns $? +function run_and_redirect +{ + local logfile=$1 + shift + + # to the log + echo "${*}" > "${logfile}" + # the actual command + "${@}" >> "${logfile}" 2>&1 +} + +## @description git patch dryrun +## @replaceable no +## @audience private +## @stability evolving +function git_dryrun +{ + local prefixsize=${1:-0} + + while [[ ${prefixsize} -lt 4 + && -z ${PATCHMODE} ]]; do + run_and_redirect "${PATCH_DIR}/apply-patch-git-dryrun.log" \ + "${GIT}" apply --binary -v --check "-p${prefixsize}" "${PATCH_FILE}" + if [[ $? == 0 ]]; then + PATCHPREFIX=${prefixsize} + PATCHMODE=git + echo "Verifying the patch:" + cat "${PATCH_DIR}/apply-patch-git-dryrun.log" + break + fi + ((prefixsize=prefixsize+1)) + done + + if [[ ${prefixsize} -eq 0 ]]; then + verify_zero "${PATCH_DIR}/apply-patch-git-dryrun.log" + if [[ $? != 0 ]]; then + PATCHMODE="" + PATCHPREFIX="" + git_dryrun 1 + fi fi +} - #first off check that all of the files do not exist - FOUND_ANY=0 - for CHECK_FILE in $(cat $TMP2) - do - if [[ -f $CHECK_FILE ]]; then - FOUND_ANY=1 +## @description patch patch dryrun +## @replaceable no +## @audience private +## @stability evolving +function patch_dryrun +{ + local prefixsize=${1:-0} + + while [[ ${prefixsize} -lt 4 + && -z ${PATCHMODE} ]]; do + run_and_redirect "${PATCH_DIR}/apply-patch-patch-dryrun.log" \ + "${PATCH}" "-p${prefixsize}" -E --dry-run < "${PATCH_FILE}" + if [[ $? == 0 ]]; then + PATCHPREFIX=${prefixsize} + PATCHMODE=patch + if [[ ${DRYRUNMODE} == true ]]; then + echo "Verifying the patch:" + cat "${PATCH_DIR}/apply-patch-patch-dryrun.log" + fi + break fi + ((prefixsize=prefixsize+1)) done - if [[ "$FOUND_ANY" = "0" ]]; then - #all of the files are new files so we have to guess where the correct place to put it is. + if [[ ${prefixsize} -eq 0 ]]; then + verify_zero "${PATCH_DIR}/apply-patch-patch-dryrun.log" + if [[ $? != 0 ]]; then + PATCHMODE="" + PATCHPREFIX="" + patch_dryrun 1 + fi + fi +} + +## @description driver for dryrun methods +## @replaceable no +## @audience private +## @stability evolving +function dryrun +{ + local method - # if all of the lines start with a/ or b/, then this is a git patch that - # was generated without --no-prefix - if ! grep -qv '^a/\|^b/' $TMP2 ; then - echo Looks like this is a git patch. Stripping a/ and b/ prefixes - echo and incrementing PLEVEL - PLEVEL=$[$PLEVEL + 1] - sed -i -e 's,^[ab]/,,' $TMP2 + for method in "${PATCHMODES[@]}"; do + if declare -f ${method}_dryrun >/dev/null; then + "${method}_dryrun" fi + if [[ -n ${PATCHMODE} ]]; then + break + fi + done - PREFIX_DIRS_AND_FILES=$(cut -d '/' -f 1 $TMP2 | sort -u) - - # if we are at the project root then nothing more to do - if [[ -d hadoop-common-project ]]; then - echo Looks like this is being run at project root + if [[ -n ${PATCHMODE} ]]; then + RESULT=0 + return 0 + fi + RESULT=1 + return 1 +} - # if all of the lines start with hadoop-common/, hadoop-hdfs/, hadoop-yarn/ or hadoop-mapreduce/, this is - # relative to the hadoop root instead of the subproject root, so we need - # to chop off another layer - elif [[ "$PREFIX_DIRS_AND_FILES" =~ ^(hadoop-common-project|hadoop-hdfs-project|hadoop-yarn-project|hadoop-mapreduce-project)$ ]]; then +## @description git patch apply +## @replaceable no +## @audience private +## @stability evolving +function git_apply +{ + echo "Applying the patch:" + run_and_redirect "${PATCH_DIR}/apply-patch-git-apply.log" \ + "${GIT}" apply --binary -v --stat --apply "-p${PATCHPREFIX}" "${PATCH_FILE}" + ${GREP} -v "^Checking" "${PATCH_DIR}/apply-patch-git-apply.log" +} - echo Looks like this is relative to project root. Increasing PLEVEL - PLEVEL=$[$PLEVEL + 1] - elif ! echo "$PREFIX_DIRS_AND_FILES" | grep -vxq 'hadoop-common-project\|hadoop-hdfs-project\|hadoop-yarn-project\|hadoop-mapreduce-project' ; then - echo Looks like this is a cross-subproject patch. Try applying from the project root - cleanup 1 +## @description patch patch apply +## @replaceable no +## @audience private +## @stability evolving +function patch_apply +{ + echo "Applying the patch:" + run_and_redirect "${PATCH_DIR}/apply-patch-patch-apply.log" \ + "${PATCH}" "-p${PATCHPREFIX}" -E < "${PATCH_FILE}" + cat "${PATCH_DIR}/apply-patch-patch-apply.log" +} + + +## @description driver for patch apply methods +## @replaceable no +## @audience private +## @stability evolving +function apply +{ + if declare -f ${PATCHMODE}_apply >/dev/null; then + "${PATCHMODE}_apply" + if [[ $? -gt 0 ]]; then + RESULT=1 + else + RESULT=0 fi + else + yetus_error "ERROR: Patching method ${PATCHMODE} does not have a way to apply patches!" + RESULT=1 fi -elif $PATCH -p1 -E --dry-run < $PATCH_FILE 2>&1 > /dev/null; then - PLEVEL=1 -elif $PATCH -p2 -E --dry-run < $PATCH_FILE 2>&1 > /dev/null; then - PLEVEL=2 -else - echo "The patch does not appear to apply with p0 to p2"; - cleanup 1; -fi +} + +trap "cleanup_and_exit 1" HUP INT QUIT TERM -# If this is a dry run then exit instead of applying the patch -if [[ -n $DRY_RUN ]]; then - cleanup 0; +setup_defaults + +parse_args "$@" + +locate_patch + +dryrun + +if [[ ${RESULT} -gt 0 ]]; then + yetus_error "ERROR: Aborting! The patch cannot be verified." + cleanup_and_exit ${RESULT} fi -echo Going to apply patch with: $PATCH -p$PLEVEL -$PATCH -p$PLEVEL -E < $PATCH_FILE +if [[ ${DRYRUNMODE} == false ]]; then + apply +fi -cleanup $? +cleanup_and_exit ${RESULT} diff --git a/dev-support/test-patch-docker/Dockerfile-endstub b/dev-support/test-patch-docker/Dockerfile-endstub new file mode 100644 index 0000000000000..aa0463ef44f81 --- /dev/null +++ b/dev-support/test-patch-docker/Dockerfile-endstub @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ADD launch-test-patch.sh /testptch/launch-test-patch.sh +RUN chmod a+rx /testptch/launch-test-patch.sh +CMD /testptch/launch-test-patch.sh \ No newline at end of file diff --git a/dev-support/test-patch-docker/Dockerfile-startstub b/dev-support/test-patch-docker/Dockerfile-startstub new file mode 100644 index 0000000000000..c49b5891c902a --- /dev/null +++ b/dev-support/test-patch-docker/Dockerfile-startstub @@ -0,0 +1,85 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM ubuntu:trusty + +WORKDIR /root + +###### +# Install common dependencies from packages +###### +RUN apt-get update && apt-get install --no-install-recommends -y \ + git curl ant make maven \ + cmake gcc g++ protobuf-compiler \ + build-essential libtool \ + zlib1g-dev pkg-config libssl-dev \ + snappy libsnappy-dev \ + bzip2 libbz2-dev \ + libjansson-dev \ + fuse libfuse-dev \ + libcurl4-openssl-dev \ + python python2.7 pylint \ + ruby \ + openjdk-7-jdk \ + libperl-critic-perl + +# Fixing the Apache commons / Maven dependency problem under Ubuntu: +# See http://wiki.apache.org/commons/VfsProblems +RUN cd /usr/share/maven/lib && ln -s ../../java/commons-lang.jar . + +####### +# Oracle Java +####### + +RUN apt-get install -y software-properties-common +RUN add-apt-repository -y ppa:webupd8team/java +RUN apt-get update + + +# Auto-accept the Oracle JDK license +RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections +RUN apt-get install -y oracle-java7-installer + +# Auto-accept the Oracle JDK license +RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections +RUN apt-get install -y oracle-java8-installer + +###### +# Install findbugs +###### +RUN mkdir -p /opt/findbugs && \ + curl https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download \ + -o /opt/findbugs.tar.gz && \ + tar xzf /opt/findbugs.tar.gz --strip-components 1 -C /opt/findbugs +ENV FINDBUGS_HOME /opt/findbugs + +#### +# Install shellcheck +#### +RUN apt-get install -y cabal-install +RUN cabal update && cabal install shellcheck --global + +#### +# Install rubocop +### +RUN gem install rubocop + +#### +# Install ruby-lint +### +RUN gem install ruby-lint + diff --git a/dev-support/test-patch-docker/launch-test-patch.sh b/dev-support/test-patch-docker/launch-test-patch.sh new file mode 100755 index 0000000000000..410149b69e3a5 --- /dev/null +++ b/dev-support/test-patch-docker/launch-test-patch.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cd "${BASEDIR}" + +if [[ -n ${JAVA_HOME} + && ! -d ${JAVA_HOME} ]]; then + echo "JAVA_HOME: ${JAVA_HOME} does not exist. Dockermode: attempting to switch to another." 1>&2 + JAVA_HOME="" +fi + +if [[ -z ${JAVA_HOME} ]]; then + JAVA_HOME=$(find /usr/lib/jvm/ -name "java-*" -type d | tail -1) + export JAVA_HOME +fi + +# Avoid out of memory errors in builds +MAVEN_OPTS=${MAVEN_OPTS:-"-Xms256m -Xmx1g"} +export MAVEN_OPTS + +# strip out --docker param to prevent re-exec again +TESTPATCHMODE=${TESTPATCHMODE/--docker } + + +cd "${BASEDIR}" +PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P) + +cd "${PATCH_DIR}/precommit/" +#shellcheck disable=SC2086 +"${PATCH_DIR}/precommit/test-patch.sh" \ + --reexec \ + --dockermode ${TESTPATCHMODE} \ + --basedir="${BASEDIR}" \ + --patch-dir="${PATCH_DIR}" \ + --java-home="${JAVA_HOME}" \ + --plugins="${PATCH_DIR}/precommit/user-plugins" \ + --jira-cmd=/opt/jiracli/jira-cli-2.2.0/jira.sh diff --git a/dev-support/test-patch-docker/test-patch-docker.sh b/dev-support/test-patch-docker/test-patch-docker.sh new file mode 100755 index 0000000000000..f8e46b4bbc183 --- /dev/null +++ b/dev-support/test-patch-docker/test-patch-docker.sh @@ -0,0 +1,383 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +DID=${RANDOM} + +## @description Print a message to stderr if --debug is turned on +## @audience private +## @stability stable +## @replaceable no +## @param string +function yetus_debug +{ + if [[ -n "${TP_SHELL_SCRIPT_DEBUG}" ]]; then + echo "[$(date) DEBUG]: $*" 1>&2 + fi +} + +## @description Run docker with some arguments, and +## @description optionally send to debug +## @audience private +## @stability evolving +## @replaceable no +## @param args +function dockercmd +{ + yetus_debug "docker $*" + docker "$@" +} + +## @description Handle command line arguments +## @audience private +## @stability evolving +## @replaceable no +## @param args +function parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --debug) + TP_SHELL_SCRIPT_DEBUG=true + ;; + --dockerversion=*) + DOCKER_VERSION=${i#*=} + ;; + --help|-help|-h|help|--h|--\?|-\?|\?) + yetus_usage + exit 0 + ;; + --java-home=*) + JAVA_HOME=${i#*=} + ;; + --patch-dir=*) + PATCH_DIR=${i#*=} + ;; + --project=*) + PROJECT_NAME=${i#*=} + ;; + *) + ;; + esac + done +} + +## @description Stop and delete all defunct containers +## @audience private +## @stability evolving +## @replaceable no +## @param args +function stop_exited_containers +{ + local line + local id + local value + local size + + echo "Docker containers in exit state:" + + dockercmd ps -a | grep Exited + + # stop *all* containers that are in exit state for + # more than > 8 hours + while read line; do + id=$(echo "${line}" | cut -f1 -d' ') + value=$(echo "${line}" | cut -f2 -d' ') + size=$(echo "${line}" | cut -f3 -d' ') + + if [[ ${size} =~ day + || ${size} =~ week + || ${size} =~ month + || ${size} =~ year ]]; then + echo "Removing docker ${id}" + dockercmd rm "${id}" + fi + + if [[ ${size} =~ hours + && ${value} -gt 8 ]]; then + echo "Removing docker ${id}" + dockercmd rm "${id}" + fi + done < <( + dockercmd ps -a \ + | grep Exited \ + | sed -e 's,ago,,g' \ + | awk '{print $1" "$(NF - 2)" "$(NF - 1)}') +} + +## @description Remove all containers that are not +## @description are not running + older than 1 day +## @audience private +## @stability evolving +## @replaceable no +## @param args +function rm_old_containers +{ + local line + local id + local value + local size + + while read line; do + id=$(echo "${line}" | cut -f1 -d, ) + state=$(echo "${line}" | cut -f2 -d, ) + stoptime=$(echo "${line}" | cut -f3 -d, | cut -f1 -d. ) + + # believe it or not, date is not even close to standardized... + if [[ $(uname -s) == Linux ]]; then + + # GNU date + stoptime=$(date -d "${stoptime}" "+%s") + else + + # BSD date + stoptime=$(date -j -f "%Y-%m-%dT%H:%M:%S" "${stoptime}" "+%s") + fi + + if [[ ${state} == false ]]; then + curtime=$(date "+%s") + ((difftime = curtime - stoptime)) + if [[ ${difftime} -gt 86400 ]]; then + echo "Removing docker ${id}" + dockercmd rm "${id}" + fi + fi + done < <( + # see https://github.com/koalaman/shellcheck/issues/375 + # shellcheck disable=SC2046 + dockercmd inspect \ + -f '{{.Id}},{{.State.Running}},{{.State.FinishedAt}}' \ + $(dockercmd ps -qa) 2>/dev/null) +} + +## @description Remove untagged/unused images +## @audience private +## @stability evolving +## @replaceable no +## @param args +function remove_untagged_images +{ + # this way is a bit more compatible with older docker versions + dockercmd images | tail -n +2 | awk '$1 == "" {print $3}' | \ + xargs --no-run-if-empty docker rmi +} + +## @description Remove defunct tagged images +## @audience private +## @stability evolving +## @replaceable no +## @param args +function remove_old_tagged_images +{ + local line + local id + local created + + while read line; do + id=$(echo "${line}" | awk '{print $1}') + created=$(echo "${line}" | awk '{print $5}') + + if [[ ${created} =~ week + || ${created} =~ month + || ${created} =~ year ]]; then + echo "Removing docker image ${id}" + dockercmd rmi "${id}" + fi + + if [[ ${id} =~ test-patch-base-${PROJECT_NAME}-date ]]; then + if [[ ${created} =~ day + || ${created} =~ hours ]]; then + echo "Removing docker image ${id}" + dockercmd rmi "${id}" + fi + fi + done < <(dockercmd images) + +} + +## @description Performance docker maintenance on Jenkins +## @audience private +## @stability evolving +## @replaceable no +## @param args +function cleanup_apache_jenkins_docker +{ + echo "==========================" + echo "Docker Images:" + dockercmd images + echo "==========================" + echo "Docker Containers:" + dockercmd ps -a + echo "==========================" + + stop_exited_containers + + rm_old_containers + + remove_untagged_images + + remove_old_tagged_images +} + +## @description Clean up our old images used for patch testing +## @audience private +## @stability evolving +## @replaceable no +## @param args +function cleanup_test_patch_images +{ + local images + local imagecount + local rmimage + local rmi + + # we always want to leave at least one of our images + # so that the whole thing doesn't have to be rebuilt. + # This also let's us purge any old images so that + # we can get fresh stuff sometimes + images=$(dockercmd images | grep --color=none "test-patch-tp-${PROJECT_NAME}" | awk '{print $1}') 2>&1 + + # shellcheck disable=SC2086 + imagecount=$(echo ${images} | tr ' ' '\n' | wc -l) + ((imagecount = imagecount - 1 )) + + # shellcheck disable=SC2086 + rmimage=$(echo ${images} | tr ' ' '\n' | tail -${imagecount}) + for rmi in ${rmimage} + do + echo "Removing image ${rmi}" + dockercmd rmi "${rmi}" + done +} + +## @description Perform pre-run maintenance to free up +## @description resources. With --jenkins, it is a lot +## @description more destructive. +## @audience private +## @stability evolving +## @replaceable no +## @param args +function cleanup +{ + if [[ ${TESTPATCHMODE} =~ jenkins ]]; then + cleanup_apache_jenkins_docker + fi + + cleanup_test_patch_images +} + +## @description Deterine the user name and user id of the user +## @description that the docker container should use +## @audience private +## @stability evolving +## @replaceable no +## @param args +function determine_user +{ + # On the Apache Jenkins hosts, $USER is pretty much untrustable beacuse some + # ... person ... sets it to an account that doesn't actually exist. + # so instead, we need to try and override it with something that's + # probably close to reality. + if [[ ${TESTPATCHMODE} =~ jenkins ]]; then + USER=$(id | cut -f2 -d\( | cut -f1 -d\)) + fi + + if [[ "$(uname -s)" == "Linux" ]]; then + USER_NAME=${SUDO_USER:=$USER} + USER_ID=$(id -u "${USER_NAME}") + GROUP_ID=$(id -g "${USER_NAME}") + else # boot2docker uid and gid + USER_NAME=${USER} + USER_ID=1000 + GROUP_ID=50 + fi +} + +## @description Determine the revision of a dockerfile +## @audience private +## @stability evolving +## @replaceable no +## @param args +function getdockerfilerev +{ + grep 'TEST_PATCH_PRIVATE: gitrev=' \ + "${PATCH_DIR}/precommit/test-patch-docker/Dockerfile" \ + | cut -f2 -d= +} + +## @description Start a test patch docker container +## @audience private +## @stability evolving +## @replaceable no +## @param args +function run_image +{ + local dockerfilerev + local baseimagename + + dockerfilerev=$(getdockerfilerev) + + baseimagename="test-patch-base-${PROJECT_NAME}-${dockerfilerev}" + + # make a base image, if it isn't available + dockercmd build -t "${baseimagename}" "${PATCH_DIR}/precommit/test-patch-docker" + + # using the base image, make one that is patch specific + dockercmd build -t "test-patch-tp-${PROJECT_NAME}-${DID}" - < The 'ant' command to use (default \${ANT_HOME}/bin/ant, or 'ant')" +} + +function ant_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --ant-cmd=*) + ANT=${i#*=} + ;; + esac + done + + # if we requested offline, pass that to mvn + if [[ ${OFFLINE} == "true" ]]; then + ANT_ARGS=(${ANT_ARGS[@]} -Doffline=) + fi +} + +function ant_buildfile +{ + echo "build.xml" +} + +function ant_executor +{ + echo "${ANT}" "${ANT_ARGS[@]}" +} + +function ant_modules_worker +{ + declare branch=$1 + declare tst=$2 + shift 2 + + case ${tst} in + javac) + modules_workers ${branch} javac + ;; + javadoc) + modules_workers ${branch} javadoc clean javadoc + ;; + unit) + modules_workers ${branch} unit + ;; + *) + yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}" + return 1 + ;; + esac +} + +function ant_count_javac_probs +{ + declare warningfile=$1 + declare val1 + declare val2 + + #shellcheck disable=SC2016 + val1=$(${GREP} -E "\[javac\] [0-9]+ errors?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}') + #shellcheck disable=SC2016 + val2=$(${GREP} -E "\[javac\] [0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}') + echo $((val1+val2)) +} + +## @description Helper for check_patch_javadoc +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function ant_count_javadoc_probs +{ + local warningfile=$1 + local val1 + local val2 + + #shellcheck disable=SC2016 + val1=$(${GREP} -E "\[javadoc\] [0-9]+ errors?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}') + #shellcheck disable=SC2016 + val2=$(${GREP} -E "\[javadoc\] [0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}') + echo $((val1+val2)) +} + +function ant_builtin_personality_file_tests +{ + local filename=$1 + + yetus_debug "Using builtin ant personality_file_tests" + + if [[ ${filename} =~ \.sh + || ${filename} =~ \.cmd + ]]; then + yetus_debug "tests/shell: ${filename}" + elif [[ ${filename} =~ \.c$ + || ${filename} =~ \.cc$ + || ${filename} =~ \.h$ + || ${filename} =~ \.hh$ + || ${filename} =~ \.proto$ + || ${filename} =~ src/test + || ${filename} =~ \.cmake$ + || ${filename} =~ CMakeLists.txt + ]]; then + yetus_debug "tests/units: ${filename}" + add_test javac + add_test unit + elif [[ ${filename} =~ build.xml + || ${filename} =~ ivy.xml + || ${filename} =~ \.java$ + ]]; then + yetus_debug "tests/javadoc+units: ${filename}" + add_test javac + add_test javadoc + add_test unit + fi + + if [[ ${filename} =~ \.java$ ]]; then + add_test findbugs + fi +} \ No newline at end of file diff --git a/dev-support/test-patch.d/asflicense.sh b/dev-support/test-patch.d/asflicense.sh new file mode 100755 index 0000000000000..27349bfdfabd3 --- /dev/null +++ b/dev-support/test-patch.d/asflicense.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [[ ${BUILDTOOL} == maven + || ${BUILDTOOL} == ant ]]; then + add_plugin asflicense + add_test asflicense +fi + +## @description Verify all files have an Apache License +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function asflicense_postapply +{ + local numpatch + + big_console_header "Determining number of patched ASF License errors" + + start_clock + + personality_modules patch asflicense + case ${BUILDTOOL} in + maven) + modules_workers patch asflicense apache-rat:check + ;; + ant) + modules_workers patch asflicense releaseaudit + ;; + *) + return 0 + ;; + esac + + # RAT fails the build if there are license problems. + # so let's take advantage of that a bit. + if [[ $? == 0 ]]; then + add_vote_table 1 asflicense "Patch does not generate ASF License warnings." + return 0 + fi + + #shellcheck disable=SC2038 + find "${BASEDIR}" -name rat.txt -o -name releaseaudit_report.txt \ + | xargs cat > "${PATCH_DIR}/patch-asflicense.txt" + + if [[ -s "${PATCH_DIR}/patch-asflicense.txt" ]] ; then + numpatch=$("${GREP}" -c '\!?????' "${PATCH_DIR}/patch-asflicense.txt") + echo "" + echo "" + echo "There appear to be ${numpatch} ASF License warnings after applying the patch." + if [[ -n ${numpatch} + && ${numpatch} -gt 0 ]] ; then + add_vote_table -1 asflicense "Patch generated ${numpatch} ASF License warnings." + + echo "Lines that start with ????? in the ASF License "\ + "report indicate files that do not have an Apache license header:" \ + > "${PATCH_DIR}/patch-asflicense-problems.txt" + + ${GREP} '\!?????' "${PATCH_DIR}/patch-asflicense.txt" \ + >> "${PATCH_DIR}/patch-asflicense-problems.txt" + + add_footer_table asflicense "@@BASE@@/patch-asflicense-problems.txt" + fi + else + # if we're here, then maven actually failed + modules_messages patch asflicense true + fi + return 1 +} diff --git a/dev-support/test-patch.d/builtin-bugsystem.sh b/dev-support/test-patch.d/builtin-bugsystem.sh new file mode 100755 index 0000000000000..9a9ee05b9b95a --- /dev/null +++ b/dev-support/test-patch.d/builtin-bugsystem.sh @@ -0,0 +1,163 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# This bug system handles the output on the screen. + +add_bugsystem console + +# we always call this one last + +function generic_locate_patch +{ + declare input=$1 + declare output=$2 + + if [[ "${OFFLINE}" == true ]]; then + yetus_debug "generic_locate_patch: offline, skipping" + return 1 + fi + + ${CURL} --silent \ + --output "${output}" \ + "${input}" + if [[ $? != 0 ]]; then + yetus_debug "jira_locate_patch: not a JIRA." + return 1 + fi + return 0 +} + +## @description Print out the finished details on the console +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +## @return 0 on success +## @return 1 on failure +function console_finalreport +{ + declare result=$1 + shift + declare i=0 + declare ourstring + declare vote + declare subs + declare ela + declare comment + declare commentfile1="${PATCH_DIR}/comment.1" + declare commentfile2="${PATCH_DIR}/comment.2" + declare normaltop + declare line + declare seccoladj=0 + declare spcfx=${PATCH_DIR}/spcl.txt + + if [[ ${result} == 0 ]]; then + if [[ ${JENKINS} == false ]]; then + { + printf "IF9fX19fX19fX18gCjwgU3VjY2VzcyEgPgogLS0tLS0tLS0tLSAKIFwgICAg"; + printf "IC9cICBfX18gIC9cCiAgXCAgIC8vIFwvICAgXC8gXFwKICAgICAoKCAgICBP"; + printf "IE8gICAgKSkKICAgICAgXFwgLyAgICAgXCAvLwogICAgICAgXC8gIHwgfCAg"; + printf "XC8gCiAgICAgICAgfCAgfCB8ICB8ICAKICAgICAgICB8ICB8IHwgIHwgIAog"; + printf "ICAgICAgIHwgICBvICAgfCAgCiAgICAgICAgfCB8ICAgfCB8ICAKICAgICAg"; + printf "ICB8bXwgICB8bXwgIAo" + } > "${spcfx}" + fi + printf "\n\n+1 overall\n\n" + else + if [[ ${JENKINS} == false ]]; then + { + printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf"; + printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf"; + printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98"; + printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg"; + printf "ICAgICAgICAgICAgICAgICAK" + } > "${spcfx}" + fi + printf "\n\n-1 overall\n\n" + fi + + if [[ -f ${spcfx} ]]; then + if which base64 >/dev/null 2>&1; then + base64 --decode "${spcfx}" 2>/dev/null + elif which openssl >/dev/null 2>&1; then + openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null + fi + echo + echo + rm "${spcfx}" + fi + + seccoladj=$(findlargest 2 "${TP_VOTE_TABLE[@]}") + if [[ ${seccoladj} -lt 10 ]]; then + seccoladj=10 + fi + + seccoladj=$((seccoladj + 2 )) + i=0 + until [[ $i -eq ${#TP_HEADER[@]} ]]; do + printf "%s\n" "${TP_HEADER[${i}]}" + ((i=i+1)) + done + + printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment" + echo "============================================================================" + i=0 + until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do + ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\|) + subs=$(echo "${ourstring}" | cut -f3 -d\|) + ela=$(echo "${ourstring}" | cut -f4 -d\|) + comment=$(echo "${ourstring}" | cut -f5 -d\|) + + echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}" + normaltop=$(head -1 "${commentfile1}") + ${SED} -e '1d' "${commentfile1}" > "${commentfile2}" + + printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \ + "${subs}" "${ela}" "${normaltop}" + while read line; do + printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}" + done < "${commentfile2}" + + ((i=i+1)) + rm "${commentfile2}" "${commentfile1}" 2>/dev/null + done + + if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then + seccoladj=$(findlargest 1 "${TP_TEST_TABLE[@]}") + printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason" + i=0 + until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do + ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\|) + subs=$(echo "${ourstring}" | cut -f3 -d\|) + printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}" + ((i=i+1)) + done + fi + + printf "\n\n|| Subsystem || Report/Notes ||\n" + echo "============================================================================" + i=0 + + until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") + printf "%s\n" "${comment}" + ((i=i+1)) + done +} diff --git a/dev-support/test-patch.d/builtin-personality.sh b/dev-support/test-patch.d/builtin-personality.sh new file mode 100755 index 0000000000000..4a720b17d60fd --- /dev/null +++ b/dev-support/test-patch.d/builtin-personality.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +function builtin_personality_modules +{ + local repostatus=$1 + local testtype=$2 + + local module + + yetus_debug "Using builtin personality_modules" + yetus_debug "Personality: ${repostatus} ${testtype}" + + clear_personality_queue + + # this always makes sure the local repo has a fresh + # copy of everything per pom rules. + if [[ ${repostatus} == branch + && ${testtype} == mvninstall ]];then + personality_enqueue_module . + return + fi + + for module in ${CHANGED_MODULES}; do + # shellcheck disable=SC2086 + personality_enqueue_module ${module} + done +} + +function personality_modules +{ + builtin_personality_modules "$@" +} + +function personality_file_tests +{ + ${BUILDTOOL}_builtin_personality_file_tests "$@" +} diff --git a/dev-support/test-patch.d/checkstyle.sh b/dev-support/test-patch.d/checkstyle.sh index 63115842b39d1..d5e216b658a67 100755 --- a/dev-support/test-patch.d/checkstyle.sh +++ b/dev-support/test-patch.d/checkstyle.sh @@ -18,188 +18,194 @@ add_plugin checkstyle CHECKSTYLE_TIMER=0 -# if it ends in an explicit .sh, then this is shell code. -# if it doesn't have an extension, we assume it is shell code too function checkstyle_filefilter { local filename=$1 - if [[ ${filename} =~ \.java$ ]]; then - add_test checkstyle + if [[ ${BUILDTOOL} == maven + || ${BUILDTOOL} == ant ]]; then + if [[ ${filename} =~ \.java$ ]]; then + add_test checkstyle + fi fi } -function checkstyle_mvnrunner +function checkstyle_runner { - local logfile=$1 - local output=$2 + local repostatus=$1 local tmp=${PATCH_DIR}/$$.${RANDOM} local j - - "${MVN}" clean test checkstyle:checkstyle -DskipTests \ - -Dcheckstyle.consoleOutput=true \ - "-D${PROJECT_NAME}PatchProcess" 2>&1 \ - | tee "${logfile}" \ - | ${GREP} ^/ \ - | ${SED} -e "s,${BASEDIR},.,g" \ - > "${tmp}" - - # the checkstyle output files are massive, so - # let's reduce the work by filtering out files - # that weren't changed. Some modules are - # MASSIVE and this can cut the output down to - # by orders of magnitude!! - for j in ${CHANGED_FILES}; do - ${GREP} "${j}" "${tmp}" >> "${output}" - done - - rm "${tmp}" 2>/dev/null -} - -function checkstyle_preapply -{ - local module_suffix - local modules=${CHANGED_MODULES} - local module - - verify_needed_test checkstyle - - if [[ $? == 0 ]]; then - return 0 + local i=0 + local fn + local savestart=${TIMER} + local savestop + local output + local logfile + local repo + local modulesuffix + local cmd + + modules_reset + + if [[ ${repostatus} == branch ]]; then + repo=${PATCH_BRANCH} + else + repo="the patch" fi - big_console_header "checkstyle plugin: prepatch" - - start_clock - - for module in ${modules} - do - pushd "${module}" >/dev/null - echo " Running checkstyle in ${module}" - module_suffix=$(basename "${module}") + #shellcheck disable=SC2153 + until [[ $i -eq ${#MODULE[@]} ]]; do + start_clock + fn=$(module_file_fragment "${MODULE[${i}]}") + modulesuffix=$(basename "${MODULE[${i}]}") + output="${PATCH_DIR}/${repostatus}-checkstyle-${fn}.txt" + logfile="${PATCH_DIR}/maven-${repostatus}-checkstyle-${fn}.txt" + pushd "${BASEDIR}/${MODULE[${i}]}" >/dev/null + + case ${BUILDTOOL} in + maven) + cmd="${MAVEN} ${MAVEN_ARGS[*]} \ + checkstyle:checkstyle \ + -Dcheckstyle.consoleOutput=true \ + ${MODULEEXTRAPARAM[${i}]//@@@MODULEFN@@@/${fn}} -Ptest-patch" + ;; + ant) + cmd="${ANT} \ + -Dcheckstyle.consoleOutput=true \ + ${MODULEEXTRAPARAM[${i}]//@@@MODULEFN@@@/${fn}} \ + ${ANT_ARGS[*]} checkstyle" + ;; + esac + + #shellcheck disable=SC2086 + echo ${cmd} "> ${logfile}" + #shellcheck disable=SC2086 + ${cmd} 2>&1 \ + | tee "${logfile}" \ + | ${GREP} ^/ \ + | ${SED} -e "s,${BASEDIR},.,g" \ + > "${tmp}" + + if [[ $? == 0 ]] ; then + module_status ${i} +1 "${logfile}" "${modulesuffix} in ${repo} passed checkstyle" + else + module_status ${i} -1 "${logfile}" "${modulesuffix} in ${repo} failed checkstyle" + ((result = result + 1)) + fi + savestop=$(stop_clock) + #shellcheck disable=SC2034 + MODULE_STATUS_TIMER[${i}]=${savestop} - checkstyle_mvnrunner \ - "${PATCH_DIR}/maven-${PATCH_BRANCH}checkstyle-${module_suffix}.txt" \ - "${PATCH_DIR}/${PATCH_BRANCH}checkstyle${module_suffix}.txt" + for j in ${CHANGED_FILES}; do + ${GREP} "${j}" "${tmp}" >> "${output}" + done - if [[ $? != 0 ]] ; then - echo "Pre-patch ${PATCH_BRANCH} checkstyle compilation is broken?" - add_jira_table -1 checkstyle "Pre-patch ${PATCH_BRANCH} ${module} checkstyle compilation may be broken." - fi + rm "${tmp}" 2>/dev/null + # shellcheck disable=SC2086 popd >/dev/null + ((i=i+1)) done - # keep track of how much as elapsed for us already - CHECKSTYLE_TIMER=$(stop_clock) + TIMER=${savestart} + + if [[ ${result} -gt 0 ]]; then + return 1 + fi return 0 } -function checkstyle_calcdiffs +function checkstyle_preapply { - local orig=$1 - local new=$2 - local diffout=$3 - local tmp=${PATCH_DIR}/cs.$$.${RANDOM} - local count=0 - local j + local result - # first, pull out just the errors - # shellcheck disable=SC2016 - ${AWK} -F: '{print $NF}' "${orig}" >> "${tmp}.branch" + big_console_header "${PATCH_BRANCH} checkstyle" - # shellcheck disable=SC2016 - ${AWK} -F: '{print $NF}' "${new}" >> "${tmp}.patch" + start_clock - # compare the errors, generating a string of line - # numbers. Sorry portability: GNU diff makes this too easy - ${DIFF} --unchanged-line-format="" \ - --old-line-format="" \ - --new-line-format="%dn " \ - "${tmp}.branch" \ - "${tmp}.patch" > "${tmp}.lined" + verify_needed_test checkstyle + if [[ $? == 0 ]]; then + echo "Patch does not need checkstyle testing." + return 0 + fi - # now, pull out those lines of the raw output - # shellcheck disable=SC2013 - for j in $(cat "${tmp}.lined"); do - # shellcheck disable=SC2086 - head -${j} "${new}" | tail -1 >> "${diffout}" - done + personality_modules branch checkstyle + checkstyle_runner branch + result=$? + modules_messages branch checkstyle true - if [[ -f "${diffout}" ]]; then - # shellcheck disable=SC2016 - count=$(wc -l "${diffout}" | ${AWK} '{print $1}' ) + # keep track of how much as elapsed for us already + CHECKSTYLE_TIMER=$(stop_clock) + if [[ ${result} != 0 ]]; then + return 1 fi - rm "${tmp}.branch" "${tmp}.patch" "${tmp}.lined" 2>/dev/null - echo "${count}" + return 0 } function checkstyle_postapply { - local rc=0 + local result local module - local modules=${CHANGED_MODULES} - local module_suffix + local fn + local i=0 local numprepatch=0 local numpostpatch=0 local diffpostpatch=0 - verify_needed_test checkstyle + big_console_header "Patch checkstyle plugin" + start_clock + + verify_needed_test checkstyle if [[ $? == 0 ]]; then + echo "Patch does not need checkstyle testing." return 0 fi - big_console_header "checkstyle plugin: postpatch" + personality_modules patch checkstyle + checkstyle_runner patch + result=$? - start_clock # add our previous elapsed to our new timer # by setting the clock back offset_clock "${CHECKSTYLE_TIMER}" - for module in ${modules} - do - pushd "${module}" >/dev/null - echo " Running checkstyle in ${module}" - module_suffix=$(basename "${module}") - - checkstyle_mvnrunner \ - "${PATCH_DIR}/maven-patchcheckstyle-${module_suffix}.txt" \ - "${PATCH_DIR}/patchcheckstyle${module_suffix}.txt" - - if [[ $? != 0 ]] ; then - ((rc = rc +1)) - echo "Post-patch checkstyle compilation is broken." - add_jira_table -1 checkstyle "Post-patch checkstyle ${module} compilation is broken." + until [[ $i -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) continue fi + module=${MODULE[$i]} + fn=$(module_file_fragment "${module}") + if [[ ! -f "${PATCH_DIR}/branch-checkstyle-${fn}.txt" ]]; then + touch "${PATCH_DIR}/branch-checkstyle-${fn}.txt" + fi + + calcdiffs "${PATCH_DIR}/branch-checkstyle-${fn}.txt" "${PATCH_DIR}/patch-checkstyle-${fn}.txt" > "${PATCH_DIR}/diff-checkstyle-${fn}.txt" #shellcheck disable=SC2016 - diffpostpatch=$(checkstyle_calcdiffs \ - "${PATCH_DIR}/${PATCH_BRANCH}checkstyle${module_suffix}.txt" \ - "${PATCH_DIR}/patchcheckstyle${module_suffix}.txt" \ - "${PATCH_DIR}/diffcheckstyle${module_suffix}.txt" ) + diffpostpatch=$(wc -l "${PATCH_DIR}/diff-checkstyle-${fn}.txt" | ${AWK} '{print $1}') if [[ ${diffpostpatch} -gt 0 ]] ; then - ((rc = rc + 1)) + ((result = result + 1)) # shellcheck disable=SC2016 - numprepatch=$(wc -l "${PATCH_DIR}/${PATCH_BRANCH}checkstyle${module_suffix}.txt" | ${AWK} '{print $1}') + numprepatch=$(wc -l "${PATCH_DIR}/branch-checkstyle-${fn}.txt" | ${AWK} '{print $1}') # shellcheck disable=SC2016 - numpostpatch=$(wc -l "${PATCH_DIR}/patchcheckstyle${module_suffix}.txt" | ${AWK} '{print $1}') + numpostpatch=$(wc -l "${PATCH_DIR}/patch-checkstyle-${fn}.txt" | ${AWK} '{print $1}') - add_jira_table -1 checkstyle "The applied patch generated "\ - "${diffpostpatch} new checkstyle issues (total was ${numprepatch}, now ${numpostpatch})." - footer="${footer} @@BASE@@/diffcheckstyle${module_suffix}.txt" + module_status ${i} -1 "diff-checkstyle-${fn}.txt" "Patch generated "\ + "${diffpostpatch} new checkstyle issues in "\ + "${module} (total was ${numprepatch}, now ${numpostpatch})." fi - - popd >/dev/null + ((i=i+1)) done - if [[ ${rc} -gt 0 ]] ; then - add_jira_footer checkstyle "${footer}" + modules_messages patch checkstyle true + + if [[ ${result} != 0 ]]; then return 1 fi - add_jira_table +1 checkstyle "There were no new checkstyle issues." return 0 -} \ No newline at end of file +} diff --git a/dev-support/test-patch.d/findbugs.sh b/dev-support/test-patch.d/findbugs.sh new file mode 100755 index 0000000000000..4fa54288bf6bd --- /dev/null +++ b/dev-support/test-patch.d/findbugs.sh @@ -0,0 +1,380 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +FINDBUGS_HOME=${FINDBUGS_HOME:-} +FINDBUGS_WARNINGS_FAIL_PRECHECK=false + +add_plugin findbugs + +function findbugs_filefilter +{ + local filename=$1 + + if [[ ${BUILDTOOL} == maven + || ${BUILDTOOL} == ant ]]; then + if [[ ${filename} =~ \.java$ + || ${filename} =~ (^|/)findbugs-exclude.xml$ ]]; then + add_test findbugs + fi + fi +} + +function findbugs_usage +{ + echo "FindBugs specific:" + echo "--findbugs-home= Findbugs home directory (default FINDBUGS_HOME environment variable)" + echo "--findbugs-strict-precheck If there are Findbugs warnings during precheck, fail" +} + +function findbugs_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --findbugs-home=*) + FINDBUGS_HOME=${i#*=} + ;; + --findbugs-strict-precheck) + FINDBUGS_WARNINGS_FAIL_PRECHECK=true + ;; + esac + done +} + +## @description are the needed bits for findbugs present? +## @audience private +## @stability evolving +## @replaceable no +## @return 0 findbugs will work for our use +## @return 1 findbugs is missing some component +function findbugs_is_installed +{ + if [[ ! -e "${FINDBUGS_HOME}/bin/findbugs" ]]; then + printf "\n\n%s is not executable.\n\n" "${FINDBUGS_HOME}/bin/findbugs" + add_vote_table -1 findbugs "Findbugs is not installed." + return 1 + fi + return 0 +} + +## @description Run the maven findbugs plugin and record found issues in a bug database +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function findbugs_runner +{ + local name=$1 + local module + local result=0 + local fn + local warnings_file + local i=0 + local savestop + + personality_modules "${name}" findbugs + case ${BUILDTOOL} in + maven) + modules_workers "${name}" findbugs test-compile findbugs:findbugs + ;; + ant) + modules_workers "${name}" findbugs findbugs + ;; + esac + + #shellcheck disable=SC2153 + until [[ ${i} -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi + start_clock + offset_clock "${MODULE_STATUS_TIMER[${i}]}" + module="${MODULE[${i}]}" + fn=$(module_file_fragment "${module}") + + case ${BUILDTOOL} in + maven) + file="${module}/target/findbugsXml.xml" + ;; + ant) + file="${ANT_FINDBUGSXML}" + ;; + esac + + if [[ ! -f ${file} ]]; then + module_status ${i} -1 "" "${name}/${module} no findbugs output file (${file})" + ((i=i+1)) + continue + fi + + warnings_file="${PATCH_DIR}/${name}-findbugs-${fn}-warnings" + + cp -p "${file}" "${warnings_file}.xml" + + if [[ ${name} == branch ]]; then + "${FINDBUGS_HOME}/bin/setBugDatabaseInfo" -name "${PATCH_BRANCH}" \ + "${warnings_file}.xml" "${warnings_file}.xml" + else + "${FINDBUGS_HOME}/bin/setBugDatabaseInfo" -name patch \ + "${warnings_file}.xml" "${warnings_file}.xml" + fi + if [[ $? != 0 ]]; then + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + module_status ${i} -1 "" "${name}/${module} cannot run setBugDatabaseInfo from findbugs" + ((result=result+1)) + ((i=i+1)) + continue + fi + + "${FINDBUGS_HOME}/bin/convertXmlToText" -html \ + "${warnings_file}.xml" \ + "${warnings_file}.html" + if [[ $? != 0 ]]; then + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + module_status ${i} -1 "" "${name}/${module} cannot run convertXmlToText from findbugs" + ((result=result+1)) + fi + + if [[ -z ${FINDBUGS_VERSION} + && ${name} == branch ]]; then + FINDBUGS_VERSION=$(${GREP} -i "BugCollection version=" "${warnings_file}.xml" \ + | cut -f2 -d\" \ + | cut -f1 -d\" ) + if [[ -n ${FINDBUGS_VERSION} ]]; then + add_footer_table findbugs "v${FINDBUGS_VERSION}" + fi + fi + + ((i=i+1)) + done + return ${result} +} + +## @description Track pre-existing findbugs warnings +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function findbugs_preapply +{ + local fn + local module + local i=0 + local warnings_file + local module_findbugs_warnings + local result=0 + + big_console_header "Pre-patch findbugs detection" + + verify_needed_test findbugs + + if [[ $? == 0 ]]; then + echo "Patch does not appear to need findbugs tests." + return 0 + fi + + findbugs_is_installed + if [[ $? != 0 ]]; then + return 1 + fi + + findbugs_runner branch + result=$? + + if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" ]]; then + until [[ $i -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi + module=${MODULE[${i}]} + start_clock + offset_clock "${MODULE_STATUS_TIMER[${i}]}" + fn=$(module_file_fragment "${module}") + warnings_file="${PATCH_DIR}/branch-findbugs-${fn}-warnings" + # shellcheck disable=SC2016 + module_findbugs_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -first \ + "${PATCH_BRANCH}" \ + "${warnings_file}.xml" \ + "${warnings_file}.xml" \ + | ${AWK} '{print $1}') + + if [[ ${module_findbugs_warnings} -gt 0 ]] ; then + module_status ${i} -1 "branch-findbugs-${fn}.html" "${module} in ${PATCH_BRANCH} cannot run convertXmlToText from findbugs" + ((result=result+1)) + fi + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + ((i=i+1)) + done + modules_messages branch findbugs true + fi + + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + +## @description Verify patch does not trigger any findbugs warnings +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function findbugs_postinstall +{ + local module + local fn + local combined_xml + local branchxml + local patchxml + local newbugsbase + local new_findbugs_warnings + local line + local firstpart + local secondpart + local i=0 + local result=0 + local savestop + + big_console_header "Patch findbugs detection" + + verify_needed_test findbugs + + if [[ $? == 0 ]]; then + echo "Patch does not appear to need findbugs tests." + return 0 + fi + + findbugs_is_installed + if [[ $? != 0 ]]; then + return 1 + fi + + findbugs_runner patch + + until [[ $i -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi + start_clock + offset_clock "${MODULE_STATUS_TIMER[${i}]}" + module="${MODULE[${i}]}" + pushd "${module}" >/dev/null + fn=$(module_file_fragment "${module}") + + combined_xml="${PATCH_DIR}/combined-findbugs-${fn}.xml" + branchxml="${PATCH_DIR}/branch-findbugs-${fn}-warnings.xml" + patchxml="${PATCH_DIR}/patch-findbugs-${fn}-warnings.xml" + + if [[ ! -f "${branchxml}" ]]; then + branchxml=${patchxml} + fi + + newbugsbase="${PATCH_DIR}/new-findbugs-${fn}" + + "${FINDBUGS_HOME}/bin/computeBugHistory" -useAnalysisTimes -withMessages \ + -output "${combined_xml}" \ + "${branchxml}" \ + "${patchxml}" + if [[ $? != 0 ]]; then + popd >/dev/null + module_status ${i} -1 "" "${module} cannot run computeBugHistory from findbugs" + ((result=result+1)) + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + ((i=i+1)) + continue + fi + + #shellcheck disable=SC2016 + new_findbugs_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -first patch \ + "${combined_xml}" "${newbugsbase}.xml" | ${AWK} '{print $1}') + if [[ $? != 0 ]]; then + popd >/dev/null + module_status ${i} -1 "" "${module} cannot run filterBugs (#1) from findbugs" + ((result=result+1)) + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + ((i=i+1)) + continue + fi + + #shellcheck disable=SC2016 + new_findbugs_fixed_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -fixed patch \ + "${combined_xml}" "${newbugsbase}.xml" | ${AWK} '{print $1}') + if [[ $? != 0 ]]; then + popd >/dev/null + module_status ${i} -1 "" "${module} cannot run filterBugs (#2) from findbugs" + ((result=result+1)) + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + ((i=i+1)) + continue + fi + + echo "Found ${new_findbugs_warnings} new Findbugs warnings and ${new_findbugs_fixed_warnings} newly fixed warnings." + findbugs_warnings=$((findbugs_warnings+new_findbugs_warnings)) + findbugs_fixed_warnings=$((findbugs_fixed_warnings+new_findbugs_fixed_warnings)) + + "${FINDBUGS_HOME}/bin/convertXmlToText" -html "${newbugsbase}.xml" \ + "${newbugsbase}.html" + if [[ $? != 0 ]]; then + popd >/dev/null + module_status ${i} -1 "" "${module} cannot run convertXmlToText from findbugs" + ((result=result+1)) + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + ((i=i+1)) + continue + fi + + if [[ ${new_findbugs_warnings} -gt 0 ]] ; then + populate_test_table FindBugs "module:${module}" + while read line; do + firstpart=$(echo "${line}" | cut -f2 -d:) + secondpart=$(echo "${line}" | cut -f9- -d' ') + add_test_table "" "${firstpart}:${secondpart}" + done < <("${FINDBUGS_HOME}/bin/convertXmlToText" "${newbugsbase}.xml") + + module_status ${i} -1 "new-findbugs-${fn}.html" "${module} introduced "\ + "${new_findbugs_warnings} new FindBugs issues." + ((result=result+1)) + fi + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${i}]=${savestop} + popd >/dev/null + ((i=i+1)) + done + + modules_messages patch findbugs true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} diff --git a/dev-support/test-patch.d/github.sh b/dev-support/test-patch.d/github.sh new file mode 100755 index 0000000000000..36c7e519ab7e0 --- /dev/null +++ b/dev-support/test-patch.d/github.sh @@ -0,0 +1,493 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This bug system provides github integration + +add_bugsystem github + +# personalities can override the following settings: + +# Web interface URL. +GITHUB_BASE_URL="https://github.com" + +# API interface URL. +GITHUB_API_URL="https://api.github.com" + +# user/repo +GITHUB_REPO="" + +# user settings +GITHUB_PASSWD="" +GITHUB_TOKEN="" +GITHUB_USER="" +GITHUB_ISSUE="" + +# private globals... +GITHUB_BRIDGED=false +GITHUB_COMMITSHA="" + +function github_usage +{ + echo "GITHUB Options:" + echo "--github-api-url= The URL of the API for github (default: '${GITHUB_API_URL}')" + echo "--github-base-url= The URL of the github server (default:'${GITHUB_BASE_URL}')" + echo "--github-password= Github password" + echo "--github-repo= github repo to use (default:'${GITHUB_REPO}')" + echo "--github-token= The token to use to write to github" + echo "--github-user= Github user" +} + +function github_parse_args +{ + declare i + + for i in "$@"; do + case ${i} in + --github-api-url=*) + GITHUB_API_URL=${i#*=} + ;; + --github-base-url=*) + GITHUB_BASE_URL=${i#*=} + ;; + --github-repo=*) + GITHUB_REPO=${i#*=} + ;; + --github-token=*) + GITHUB_TOKEN=${i#*=} + ;; + --github-password=*) + GITHUB_PASSWD=${i#*=} + ;; + --github-user=*) + GITHUB_USER=${i#*=} + ;; + esac + done +} + +## @description this gets called when JIRA thinks this +## @description issue is just a pointer to github +## @description WARNING: Called from JIRA plugin! +function github_jira_bridge +{ + declare fileloc=$1 + declare urlfromjira + + # we use this to prevent loops later on + GITHUB_BRIDGED=true + + # the JIRA issue has already been downloaded. So let's + # find the URL. This is currently hard-coded to github.com + # Sorry Github Enterprise users. :( + + # shellcheck disable=SC2016 + urlfromjira=$(${AWK} 'match($0,"https://github.com/.*patch"){print $1}' "${PATCH_DIR}/jira" | tail -1) + github_breakup_url "${urlfromjira}" + github_locate_patch "${GITHUB_ISSUE}" "${fileloc}" +} + +## @description given a URL, break it up into github plugin globals +## @description this will *override* any personality or yetus defaults +## @params url +function github_breakup_url +{ + declare url=$1 + declare count + declare pos1 + declare pos2 + + count=${url//[^\/]} + count=${#count} + ((pos2=count-3)) + ((pos1=pos2)) + + GITHUB_BASE_URL=$(echo "${url}" | cut -f1-${pos2} -d/) + + ((pos1=pos1+1)) + ((pos2=pos1+1)) + + GITHUB_REPO=$(echo "${url}" | cut -f${pos1}-${pos2} -d/) + + ((pos1=pos2+2)) + unset pos2 + + GITHUB_ISSUE=$(echo "${url}" | cut -f${pos1}-${pos2} -d/ | cut -f1 -d.) +} + + +## @description based upon a github PR, attempt to link back to JIRA +function github_find_jira_title +{ + declare title + declare maybe + declare retval + + if [[ ! -f "${PATCH_DIR}/github-pull.json" ]]; then + return 1 + fi + + title=$(GREP title "${PATCH_DIR}/github-pull.json" \ + | cut -f4 -d\") + + # people typically do two types: JIRA-ISSUE: and [JIRA-ISSUE] + # JIRA_ISSUE_RE is pretty strict so we need to chop that stuff + # out first + + maybe=$(echo "${title}" | cut -f2 -d\[ | cut -f1 -d\]) + jira_determine_issue "${maybe}" + retval=$? + + if [[ ${retval} == 0 ]]; then + return 0 + fi + + maybe=$(echo "${title}" | cut -f1 -d:) + jira_determine_issue "${maybe}" + retval=$? + + if [[ ${retval} == 0 ]]; then + return 0 + fi +} + +function github_determine_issue +{ + declare input=$1 + + if [[ ${input} =~ ^[0-9]+$ + && -n ${GITHUB_REPO} ]]; then + # shellcheck disable=SC2034 + ISSUE=${input} + if [[ -z ${GITHUB_ISSUE} ]]; then + GITHUB_ISSUE=${input} + fi + fi + + # if JIRA didn't call us, should we call it? + if [[ ${GITHUB_BRIDGED} == false ]]; then + github_find_jira_title + if [[ $? == 0 ]]; then + return 0 + fi + fi + + if [[ -n ${GITHUB_ISSUE} ]]; then + return 0 + fi + + return 1 +} + +## @description Try to guess the branch being tested using a variety of heuristics +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success, with PATCH_BRANCH updated appropriately +## @return 1 on failure +function github_determine_branch +{ + if [[ ! -f "${PATCH_DIR}/github-pull.json" ]]; then + return 1 + fi + + # shellcheck disable=SC2016 + PATCH_BRANCH=$(${AWK} 'match($0,"\"ref\": \""){print $2}' "${PATCH_DIR}/github-pull.json"\ + | cut -f2 -d\"\ + | tail -1 ) + + yetus_debug "Github determine branch: starting with ${PATCH_BRANCH}" + + verify_valid_branch "${PATCH_BRANCH}" + if [[ $? == 0 ]]; then + return 0 + fi + return 1 +} + +function github_locate_patch +{ + declare input=$1 + declare output=$2 + declare githubauth + + if [[ "${OFFLINE}" == true ]]; then + yetus_debug "github_locate_patch: offline, skipping" + return 1 + fi + + + # https://github.com/your/repo/pull/## + if [[ ${input} =~ ^${GITHUB_BASE_URL}.*/pull/[0-9]+$ ]]; then + github_breakup_url "${input}.patch" + input=${GITHUB_ISSUE} + fi + + # https://github.com/your/repo/pulls/##.patch + if [[ ${input} =~ ^${GITHUB_BASE_URL}.*patch$ ]]; then + github_breakup_url "${input}" + input=${GITHUB_ISSUE} + fi + + # https://github.com/your/repo/pulls/##.diff + if [[ ${input} =~ ^${GITHUB_BASE_URL}.*diff$ ]]; then + github_breakup_url "${input}" + input=${GITHUB_ISSUE} + fi + + # if it isn't a number at this point, no idea + # how to process + if [[ ! ${input} =~ ^[0-9]+$ ]]; then + yetus_debug "github: ${input} is not a pull request #" + return 1 + fi + + # we always pull the .patch version (even if .diff was given) + # with the assumption that this way binary files work. + # The downside of this is that the patch files are + # significantly larger and therefore take longer to process + PATCHURL="${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}.patch" + echo "GITHUB PR #${input} is being downloaded at $(date) from" + echo "${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}" + + if [[ -n "${GITHUB_USER}" + && -n "${GITHUB_PASSWD}" ]]; then + githubauth="${GITHUB_USER}:${GITHUB_PASSWD}" + elif [[ -n "${GITHUB_TOKEN}" ]]; then + githubauth="Authorization: token ${GITHUB_TOKEN}" + else + githubauth="X-ignore-me: fake" + fi + + # Let's pull the PR JSON for later use + ${CURL} --silent --fail \ + -H "Accept: application/vnd.github.v3.full+json" \ + -H "${githubauth}" \ + --output "${PATCH_DIR}/github-pull.json" \ + --location \ + "${GITHUB_API_URL}/repos/${GITHUB_REPO}/pulls/${input}" + + echo "Patch from GITHUB PR #${input} is being downloaded at $(date) from" + echo "${PATCHURL}" + + # the actual patch file + ${CURL} --silent --fail \ + --output "${output}" \ + --location \ + -H "${githubauth}" \ + "${PATCHURL}" + + if [[ $? != 0 ]]; then + yetus_debug "github_locate_patch: not a github pull request." + return 1 + fi + + GITHUB_ISSUE=${input} + + # github will translate this to be #(xx) ! + add_footer_table "GITHUB PR" "${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}" + + return 0 +} + +function github_linecomments +{ + declare plugin=$1 + declare file=$2 + # shellcheck disable=SC2034 + declare realline=$3 + declare uniline=$4 + declare text=$5 + declare tempfile="${PATCH_DIR}/ghcomment.$$.${RANDOM}" + declare githubauth + + if [[ "${file}" =~ ^./ ]]; then + file=${file##./} + fi + + if [[ -z "${GITHUB_COMMITSHA}" ]]; then + GITHUB_COMMITSHA=$(${GREP} \"sha\" "${PATCH_DIR}/github-pull.json" 2>/dev/null \ + | head -1 \ + | cut -f4 -d\") + fi + + if [[ -z "${uniline}" ]]; then + return + fi + + # build our REST post + { + printf "{\"body\":\"" + echo "${plugin}: ${text}" \ + | ${SED} -e 's,\\,\\\\,g' \ + -e 's,\",\\\",g' \ + -e 's,$,\\r\\n,g' \ + | tr -d '\n' + echo "\"," + echo "\"commit_id\":\"${GITHUB_COMMITSHA}\"," + echo "\"path\":\"${file}\"," + echo "\"position\":${uniline}" + echo "}" + } > "${tempfile}" + + if [[ -n "${GITHUB_USER}" + && -n "${GITHUB_PASSWD}" ]]; then + githubauth="${GITHUB_USER}:${GITHUB_PASSWD}" + elif [[ -n "${GITHUB_TOKEN}" ]]; then + githubauth="Authorization: token ${GITHUB_TOKEN}" + else + return 0 + fi + + ${CURL} -X POST \ + -H "Accept: application/vnd.github.v3.full+json" \ + -H "Content-Type: application/json" \ + -H "${githubauth}" \ + -d @"${tempfile}" \ + --silent --location \ + "${GITHUB_API_URL}/repos/${GITHUB_REPO}/pulls/${GITHUB_ISSUE}/comments" \ + >/dev/null + rm "${tempfile}" +} + +## @description Write the contents of a file to github +## @params filename +## @stability stable +## @audience public +function github_write_comment +{ + declare -r commentfile=${1} + declare retval=0 + declare restfile="${PATCH_DIR}/ghcomment.$$" + declare githubauth + + if [[ "${OFFLINE}" == true ]]; then + echo "Github Plugin: Running in offline, comment skipped." + return 0 + fi + + { + printf "{\"body\":\"" + ${SED} -e 's,\\,\\\\,g' \ + -e 's,\",\\\",g' \ + -e 's,$,\\r\\n,g' "${commentfile}" \ + | tr -d '\n' + echo "\"}" + } > "${restfile}" + + if [[ -n "${GITHUB_USER}" + && -n "${GITHUB_PASSWD}" ]]; then + githubauth="${GITHUB_USER}:${GITHUB_PASSWD}" + elif [[ -n "${GITHUB_TOKEN}" ]]; then + githubauth="Authorization: token ${GITHUB_TOKEN}" + else + echo "Github Plugin: no credentials provided to write a comment." + return 0 + fi + + ${CURL} -X POST \ + -H "Accept: application/vnd.github.v3.full+json" \ + -H "Content-Type: application/json" \ + -H "${githubauth}" \ + -d @"${restfile}" \ + --silent --location \ + "${GITHUB_API_URL}/repos/${GITHUB_REPO}/issues/${GITHUB_ISSUE}/comments" \ + >/dev/null + + retval=$? + rm "${restfile}" + return ${retval} +} + +## @description Print out the finished details to the Github PR +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +function github_finalreport +{ + declare result=$1 + declare i + declare commentfile=${PATCH_DIR}/gitcommentfile.$$ + declare comment + + rm "${commentfile}" 2>/dev/null + + if [[ ${JENKINS} != "true" + || -z ${GITHUB_ISSUE} ]] ; then + return 0 + fi + + big_console_header "Adding comment to Github" + + add_footer_table "Console output" "${BUILD_URL}console" + + if [[ ${result} == 0 ]]; then + echo ":confetti_ball: **+1 overall**" >> "${commentfile}" + else + echo ":broken_heart: **-1 overall**" >> "${commentfile}" + fi + + printf "\n\n\n\n" >> "${commentfile}" + + i=0 + until [[ ${i} -eq ${#TP_HEADER[@]} ]]; do + printf "%s\n\n" "${TP_HEADER[${i}]}" >> "${commentfile}" + ((i=i+1)) + done + + { + printf "\n\n" + echo "| Vote | Subsystem | Runtime | Comment |" + echo "|:----:|----------:|--------:|:--------|" + } >> "${commentfile}" + + i=0 + until [[ ${i} -eq ${#TP_VOTE_TABLE[@]} ]]; do + echo "${TP_VOTE_TABLE[${i}]}" >> "${commentfile}" + ((i=i+1)) + done + + if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then + { + printf "\n\n" + echo "| Reason | Tests |" + echo "|-------:|:------|" + } >> "${commentfile}" + i=0 + until [[ ${i} -eq ${#TP_TEST_TABLE[@]} ]]; do + echo "${TP_TEST_TABLE[${i}]}" >> "${commentfile}" + ((i=i+1)) + done + fi + + { + printf "\n\n" + echo "| Subsystem | Report/Notes |" + echo "|----------:|:-------------|" + } >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${BUILD_URL}artifact/patchprocess,g") + printf "%s\n" "${comment}" >> "${commentfile}" + ((i=i+1)) + done + + printf "\n\nThis message was automatically generated.\n\n" >> "${commentfile}" + + github_write_comment "${commentfile}" +} diff --git a/dev-support/test-patch.d/gradle.sh b/dev-support/test-patch.d/gradle.sh new file mode 100755 index 0000000000000..06947b4dc085b --- /dev/null +++ b/dev-support/test-patch.d/gradle.sh @@ -0,0 +1,221 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [[ -z "${GRADLE:-}" ]]; then + GRADLE=gradle +fi + +if [[ -z "${GRADLEW:-}" ]]; then + GRADLEW=./gradlew +fi + +add_build_tool gradle + + +declare -a GRADLE_ARGS=() + +function gradle_usage +{ + echo "gradle specific:" + echo "--gradle-cmd= The 'gradle' command to use (default 'gradle')" +} + +function gradle_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --gradle-cmd=*) + GRADLE=${i#*=} + ;; + esac + done +} + + +function gradle_buildfile +{ + echo "gradlew" +} + +function gradle_executor +{ + echo "${GRADLEW}" "${GRADLE_ARGS[@]}" +} + +## @description Bootstrap gradle +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function gradle_precheck_install +{ + local result=0 + + if [[ ${BUILDTOOL} != gradle ]]; then + return 0 + fi + + pushd "${BASEDIR}" >/dev/null + echo_and_redirect ${PATCH_DIR}/branch-gradle-bootstrap.txt gradle -b bootstrap.gradle + popd >/dev/null + + personality_modules branch gradleboot + modules_workers branch gradleboot + result=$? + modules_messages branch gradleboot true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + + +## @description Bootstrap gradle +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function gradle_postapply_install +{ + local result=0 + + if [[ ${BUILDTOOL} != gradle ]]; then + return 0 + fi + + pushd "${BASEDIR}" >/dev/null + echo_and_redirect "${PATCH_DIR}/patch-gradle-bootstrap.txt" gradle -b bootstrap.gradle + popd >/dev/null + + personality_modules patch gradleboot + modules_workers patch gradleboot + result=$? + modules_messages patch gradleboot true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + +function gradle_count_javac_probs +{ + echo 0 +} + +function gradle_count_javadoc_probs +{ + local warningfile=$1 + + #shellcheck disable=SC2016,SC2046 + ${GREP} -E "^[0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$1} END {print sum}' +} + +## @description Helper for check_patch_javadoc +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function gradle_count_scaladoc_probs +{ + local warningfile=$1 + + #shellcheck disable=SC2016,SC2046 + ${GREP} "^\[ant:scaladoc\]" "${warningfile}" | wc -l +} + +function gradle_modules_worker +{ + declare branch=$1 + declare tst=$2 + shift 2 + + case ${tst} in + checkstyle) + modules_workers ${branch} ${tst} checkstyleMain checkstyleTest + ;; + javac|scalac) + modules_workers ${branch} ${tst} clean build + ;; + javadoc) + modules_workers ${branch} javadoc javadoc + ;; + scaladoc) + modules_workers ${branch} scaladoc scaladoc + ;; + unit) + modules_workers ${branch} unit test + ;; + *) + yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}" + return 1 + ;; + esac +} + +function gradle_builtin_personality_file_tests +{ + local filename=$1 + + yetus_debug "Using builtin gradle personality_file_tests" + + if [[ ${filename} =~ src/main/webapp ]]; then + yetus_debug "tests/webapp: ${filename}" + elif [[ ${filename} =~ \.sh + || ${filename} =~ \.cmd + || ${filename} =~ src/main/scripts + || ${filename} =~ src/test/scripts + ]]; then + yetus_debug "tests/shell: ${filename}" + elif [[ ${filename} =~ \.c$ + || ${filename} =~ \.cc$ + || ${filename} =~ \.h$ + || ${filename} =~ \.hh$ + || ${filename} =~ \.proto$ + || ${filename} =~ \.cmake$ + || ${filename} =~ CMakeLists.txt + ]]; then + yetus_debug "tests/units: ${filename}" + add_test cc + add_test unit + elif [[ ${filename} =~ \.scala$ ]]; then + add_test scalac + add_test scaladoc + add_test unit + elif [[ ${filename} =~ build.xml$ + || ${filename} =~ pom.xml$ + || ${filename} =~ \.java$ + || ${filename} =~ src/main + ]]; then + yetus_debug "tests/javadoc+units: ${filename}" + add_test javac + add_test javadoc + add_test unit + fi + + if [[ ${filename} =~ src/test ]]; then + yetus_debug "tests" + add_test unit + fi + + if [[ ${filename} =~ \.java$ ]]; then + add_test findbugs + fi +} \ No newline at end of file diff --git a/dev-support/test-patch.d/jira.sh b/dev-support/test-patch.d/jira.sh new file mode 100755 index 0000000000000..d708c6c37b1bd --- /dev/null +++ b/dev-support/test-patch.d/jira.sh @@ -0,0 +1,423 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# this bug system handles JIRA. Personalities +# can override the following variables: + +# base JIRA URL +JIRA_URL=${JIRA_URL:-"https://issues.apache.org/jira"} + +# Issue regex to help identify the project +JIRA_ISSUE_RE='' + +add_bugsystem jira + +function jira_usage +{ + echo "JIRA Options:" + echo "--jira-issue-re= Bash regular expression to use when trying to find a jira ref in the patch name (default: \'${JIRA_ISSUE_RE}\')" + echo "--jira-password= The password for the 'jira' command" + echo "--jira-base-url= The URL of the JIRA server (default:'${JIRA_URL}')" + echo "--jira-user= The user for the 'jira' command" +} + +function jira_parse_args +{ + declare i + + for i in "$@"; do + case ${i} in + --jira-base-url=*) + JIRA_URL=${i#*=} + ;; + --jira-issue-re=*) + JIRA_ISSUE_RE=${i#*=} + ;; + --jira-password=*) + JIRA_PASSWD=${i#*=} + ;; + --jira-user=*) + JIRA_USER=${i#*=} + ;; + esac + done +} + +## @description provides issue determination based upon the URL and more. +## @description WARNING: called from the github plugin! +function jira_determine_issue +{ + declare input=$1 + declare patchnamechunk + declare maybeissue + + # shellcheck disable=SC2016 + patchnamechunk=$(echo "${input}" | ${AWK} -F/ '{print $NF}') + + maybeissue=$(echo "${patchnamechunk}" | cut -f1,2 -d-) + + if [[ ${maybeissue} =~ ${JIRA_ISSUE_RE} ]]; then + ISSUE=${maybeissue} + JIRA_ISSUE=${maybeissue} + add_footer_table "JIRA Issue" "${ISSUE}" + return 0 + fi + + return 1 +} + +function jira_http_fetch +{ + declare input=$1 + declare output=$2 + + if [[ -n "${JIRA_USER}" + && -n "${JIRA_PASSWD}" ]]; then + ${CURL} --silent --fail \ + --user "${JIRA_USER}:${JIRA_PASSWD}" \ + --output "${output}" \ + --location \ + "${JIRA_URL}/${input}" + else + ${CURL} --silent --fail \ + --output "${output}" \ + --location \ + "${JIRA_URL}/${input}" + fi +} + +function jira_locate_patch +{ + declare input=$1 + declare fileloc=$2 + declare relativeurl + + yetus_debug "jira_locate_patch: trying ${JIRA_URL}/browse/${input}" + + if [[ "${OFFLINE}" == true ]]; then + yetus_debug "jira_locate_patch: offline, skipping" + return 1 + fi + + jira_http_fetch "browse/${input}" "${PATCH_DIR}/jira" + + if [[ $? != 0 ]]; then + yetus_debug "jira_locate_patch: not a JIRA." + return 1 + fi + + # if github is configured and we see what looks like a URL, + # send this to the github plugin to process. + if [[ -n "${GITHUB_BASE_URL}" + && $(${GREP} -c "${GITHUB_BASE_URL}"'[^ ]*patch' "${PATCH_DIR}/jira") != 0 ]]; then + echo "${input} appears to be a Github PR. Switching Modes." + github_jira_bridge "${fileloc}" + return $? + elif [[ $(${GREP} -c 'Patch Available' "${PATCH_DIR}/jira") == 0 ]]; then + if [[ ${JENKINS} == true ]]; then + yetus_error "ERROR: ${input} is not \"Patch Available\"." + cleanup_and_exit 1 + else + yetus_error "WARNING: ${input} is not \"Patch Available\"." + fi + fi + + #shellcheck disable=SC2016 + relativeurl=$(${AWK} 'match($0,"/secure/attachment/[0-9]*/[^\"]*"){print substr($0,RSTART,RLENGTH)}' "${PATCH_DIR}/jira" | + ${GREP} -v -e 'htm[l]*$' | sort | tail -1 | ${SED} -e 's,[ ]*$,,g') + PATCHURL="${JIRA_URL}${relativeurl}" + if [[ ! ${PATCHURL} =~ \.patch$ ]]; then + guess_patch_file "${PATCH_DIR}/patch" + if [[ $? == 0 ]]; then + yetus_debug "The patch ${PATCHURL} was not named properly, but it looks like a patch file. Proceeding, but issue/branch matching might go awry." + add_vote_table 0 patch "The patch file was not named according to ${PROJECT_NAME}'s naming conventions. Please see ${HOW_TO_CONTRIBUTE} for instructions." + fi + fi + echo "${input} patch is being downloaded at $(date) from" + echo "${PATCHURL}" + add_footer_table "JIRA Patch URL" "${PATCHURL}" + jira_http_fetch "${relativeurl}" "${fileloc}" + if [[ $? != 0 ]];then + yetus_error "ERROR: ${input}/${PATCHURL} could not be downloaded." + cleanup_and_exit 1 + fi + return 0 +} + +## @description Try to guess the branch being tested using a variety of heuristics +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success, with PATCH_BRANCH updated appropriately +function jira_determine_branch +{ + declare patchnamechunk + declare total + declare count + declare hinttype + + for hinttype in "${PATCHURL}" "${PATCH_OR_ISSUE}"; do + if [[ -z "${hinttype}" ]]; then + continue + fi + + # If one of these matches the JIRA issue regex + # then we don't want it to trigger the branch + # detection since that's almost certainly not + # intended. In other words, if ISSUE-99 is the + # name of a branch, you want to test ISSUE-99 + # against master, not ISSUE-99's branch + if [[ ${hinttype} =~ ${JIRA_ISSUE_RE} ]]; then + continue + fi + + yetus_debug "Determine branch: starting with ${hinttype}" + patchnamechunk=$(echo "${hinttype}" \ + | ${SED} -e 's,.*/\(.*\)$,\1,' \ + -e 's,\.txt,.,' \ + -e 's,.patch,.,g' \ + -e 's,.diff,.,g' \ + -e 's,\.\.,.,g' \ + -e 's,\.$,,g' ) + + # ISSUE-branch-## + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1,2 -d-) + yetus_debug "Determine branch: ISSUE-branch-## = ${PATCH_BRANCH}" + if [[ -n "${PATCH_BRANCH}" ]]; then + verify_valid_branch "${PATCH_BRANCH}" + if [[ $? == 0 ]]; then + return 0 + fi + fi + + # ISSUE-##[.##].branch + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d. ) + count="${PATCH_BRANCH//[^.]}" + total=${#count} + ((total = total + 3 )) + until [[ ${total} -lt 2 ]]; do + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3-${total} -d.) + yetus_debug "Determine branch: ISSUE[.##].branch = ${PATCH_BRANCH}" + ((total=total-1)) + if [[ -n "${PATCH_BRANCH}" ]]; then + verify_valid_branch "${PATCH_BRANCH}" + if [[ $? == 0 ]]; then + return 0 + fi + fi + done + + # ISSUE.branch.## + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2- -d. ) + count="${PATCH_BRANCH//[^.]}" + total=${#count} + ((total = total + 3 )) + until [[ ${total} -lt 2 ]]; do + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2-${total} -d.) + yetus_debug "Determine branch: ISSUE.branch[.##] = ${PATCH_BRANCH}" + ((total=total-1)) + if [[ -n "${PATCH_BRANCH}" ]]; then + verify_valid_branch "${PATCH_BRANCH}" + if [[ $? == 0 ]]; then + return 0 + fi + fi + done + + # ISSUE-branch.## + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1- -d. ) + count="${PATCH_BRANCH//[^.]}" + total=${#count} + ((total = total + 1 )) + until [[ ${total} -eq 1 ]]; do + PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1-${total} -d. ) + yetus_debug "Determine branch: ISSUE-branch[.##] = ${PATCH_BRANCH}" + ((total=total-1)) + if [[ -n "${PATCH_BRANCH}" ]]; then + verify_valid_branch "${PATCH_BRANCH}" + if [[ $? == 0 ]]; then + return 0 + fi + fi + done + done + + return 1 +} + +## @description Write the contents of a file to JIRA +## @params filename +## @stability stable +## @audience public +## @returns ${CURL} exit code +function jira_write_comment +{ + declare -r commentfile=${1} + declare retval=0 + + if [[ "${OFFLINE}" == true ]]; then + echo "JIRA Plugin: Running in offline, comment skipped." + return 0 + fi + + if [[ -n ${JIRA_PASSWD} + && -n ${JIRA_USER} ]]; then + + # RESTify the comment + { + echo "{\"body\":\"" + ${SED} -e 's,\\,\\\\,g' \ + -e 's,\",\\\",g' \ + -e 's,$,\\r\\n,g' "${commentfile}" \ + | tr -d '\n' + echo "\"}" + } > "${PATCH_DIR}/jiracomment.$$" + + ${CURL} -X POST \ + -H "Accept: application/json" \ + -H "Content-Type: application/json" \ + -u "${JIRA_USER}:${JIRA_PASSWD}" \ + -d @"${PATCH_DIR}/jiracomment.$$" \ + --silent --location \ + "${JIRA_URL}/rest/api/2/issue/${JIRA_ISSUE}/comment" \ + >/dev/null + retval=$? + rm "${PATCH_DIR}/jiracomment.$$" + else + echo "JIRA Plugin: no credentials provided to write a comment." + fi + return ${retval} +} + +## @description Print out the finished details to the JIRA issue +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +function jira_finalreport +{ + declare result=$1 + declare i + declare commentfile=${PATCH_DIR}/jiracommentfile + declare comment + declare vote + declare ourstring + declare ela + declare subs + declare color + declare comment + + rm "${commentfile}" 2>/dev/null + + if [[ ${JENKINS} == "false" + || ${OFFLINE} == true ]] ; then + return 0 + fi + + if [[ -z "${JIRA_ISSUE}" ]]; then + return 0 + fi + + big_console_header "Adding comment to JIRA" + + add_footer_table "Console output" "${BUILD_URL}console" + + if [[ ${result} == 0 ]]; then + echo "| (/) *{color:green}+1 overall{color}* |" >> "${commentfile}" + else + echo "| (x) *{color:red}-1 overall{color}* |" >> "${commentfile}" + fi + + echo "\\\\" >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_HEADER[@]} ]]; do + printf "%s\n" "${TP_HEADER[${i}]}" >> "${commentfile}" + ((i=i+1)) + done + + echo "\\\\" >> "${commentfile}" + + echo "|| Vote || Subsystem || Runtime || Comment ||" >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do + ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\| | tr -d ' ') + subs=$(echo "${ourstring}" | cut -f3 -d\|) + ela=$(echo "${ourstring}" | cut -f4 -d\|) + comment=$(echo "${ourstring}" | cut -f5 -d\|) + + # summary line + if [[ -z ${vote} + && -n ${ela} ]]; then + color="black" + elif [[ -z ${vote} ]]; then + # keep same color + true + else + # new vote line + case ${vote} in + 1|"+1") + color="green" + ;; + -1) + color="red" + ;; + 0) + color="blue" + ;; + *) + color="black" + ;; + esac + fi + + printf "| {color:%s}%s{color} | {color:%s}%s{color} | {color:%s}%s{color} | {color:%s}%s{color} |\n" \ + "${color}" "${vote}" \ + "${color}" "${subs}" \ + "${color}" "${ela}" \ + "${color}" "${comment}" \ + >> "${commentfile}" + ((i=i+1)) + done + + if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then + { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" + + echo "|| Reason || Tests ||" >> "${commentfile}" + i=0 + until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do + printf "%s\n" "${TP_TEST_TABLE[${i}]}" >> "${commentfile}" + ((i=i+1)) + done + fi + + { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" + + echo "|| Subsystem || Report/Notes ||" >> "${commentfile}" + i=0 + until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${BUILD_URL}artifact/patchprocess,g") + printf "%s\n" "${comment}" >> "${commentfile}" + ((i=i+1)) + done + + printf "\n\nThis message was automatically generated.\n\n" >> "${commentfile}" + + jira_write_comment "${commentfile}" +} diff --git a/dev-support/test-patch.d/junit.sh b/dev-support/test-patch.d/junit.sh new file mode 100755 index 0000000000000..4393511b9c1be --- /dev/null +++ b/dev-support/test-patch.d/junit.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_test_format junit + +JUNIT_TEST_TIMEOUTS="" +JUNIT_FAILED_TESTS="" + +function junit_process_tests +{ + # shellcheck disable=SC2034 + declare module=$1 + declare buildlogfile=$2 + declare result=0 + declare module_test_timeouts + declare module_failed_tests + + # shellcheck disable=SC2016 + module_test_timeouts=$(${AWK} '/^Running / { array[$NF] = 1 } /^Tests run: .* in / { delete array[$NF] } END { for (x in array) { print x } }' "${buildlogfile}") + if [[ -n "${module_test_timeouts}" ]] ; then + JUNIT_TEST_TIMEOUTS="${JUNIT_TEST_TIMEOUTS} ${module_test_timeouts}" + ((result=result+1)) + fi + + #shellcheck disable=SC2026,SC2038,SC2016 + module_failed_tests=$(find . -name 'TEST*.xml'\ + | xargs "${GREP}" -l -E " The 'mvn' command to use (default \${MAVEN_HOME}/bin/mvn, or 'mvn')" +} + +function maven_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --mvn-cmd=*) + MAVEN=${i#*=} + ;; + esac + done + + if [[ ${OFFLINE} == "true" ]]; then + MAVEN_ARGS=(${MAVEN_ARGS[@]} --offline) + fi +} + +function maven_buildfile +{ + echo "pom.xml" +} + +function maven_executor +{ + echo "${MAVEN}" "${MAVEN_ARGS[@]}" +} + +function mvnsite_filefilter +{ + local filename=$1 + + if [[ ${BUILDTOOL} = maven ]]; then + if [[ ${filename} =~ src/site ]]; then + yetus_debug "tests/mvnsite: ${filename}" + add_test mvnsite + fi + fi +} + +function maven_modules_worker +{ + declare branch=$1 + declare tst=$2 + + case ${tst} in + javac|scalac) + modules_workers ${branch} ${tst} clean test-compile + ;; + javadoc) + modules_workers ${branch} javadoc clean javadoc:javadoc + ;; + unit) + modules_workers ${branch} unit clean test -fae + ;; + *) + yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}" + return 1 + ;; + esac +} + +function maven_count_javac_probs +{ + local warningfile=$1 + + #shellcheck disable=SC2016,SC2046 + ${GREP} '\[WARNING\]' "${warningfile}" | ${AWK} '{sum+=1} END {print sum}' +} + +function maven_builtin_personality_file_tests +{ + local filename=$1 + + yetus_debug "Using builtin mvn personality_file_tests" + + if [[ ${filename} =~ src/main/webapp ]]; then + yetus_debug "tests/webapp: ${filename}" + elif [[ ${filename} =~ \.sh + || ${filename} =~ \.cmd + || ${filename} =~ src/main/scripts + || ${filename} =~ src/test/scripts + ]]; then + yetus_debug "tests/shell: ${filename}" + elif [[ ${filename} =~ \.c$ + || ${filename} =~ \.cc$ + || ${filename} =~ \.h$ + || ${filename} =~ \.hh$ + || ${filename} =~ \.proto$ + || ${filename} =~ \.cmake$ + || ${filename} =~ CMakeLists.txt + ]]; then + yetus_debug "tests/units: ${filename}" + add_test cc + add_test unit + elif [[ ${filename} =~ \.scala$ ]]; then + add_test javac + add_test scaladoc + add_test unit + elif [[ ${filename} =~ build.xml$ + || ${filename} =~ pom.xml$ + || ${filename} =~ \.java$ + || ${filename} =~ src/main + ]]; then + yetus_debug "tests/javadoc+units: ${filename}" + add_test javac + add_test javadoc + add_test unit + fi + + if [[ ${filename} =~ src/test ]]; then + yetus_debug "tests" + add_test unit + fi + + if [[ ${filename} =~ \.java$ ]]; then + add_test findbugs + fi +} + +## @description Helper for check_patch_javadoc +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function maven_count_javadoc_probs +{ + local warningfile=$1 + + #shellcheck disable=SC2016,SC2046 + ${GREP} -E "^[0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$1} END {print sum}' +} + +## @description Confirm site pre-patch +## @audience private +## @stability stable +## @replaceable no +## @return 0 on success +## @return 1 on failure +function mvnsite_preapply +{ + local result=0 + + if [[ ${BUILDTOOL} != maven ]]; then + return 0 + fi + + verify_needed_test mvnsite + if [[ $? == 0 ]];then + return 0 + fi + big_console_header "Pre-patch ${PATCH_BRANCH} site verification" + + + personality_modules branch mvnsite + modules_workers branch mvnsite clean site site:stage + result=$? + modules_messages branch mvnsite true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + +## @description Make sure site still compiles +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function mvnsite_postapply +{ + local result=0 + + if [[ ${BUILDTOOL} != maven ]]; then + return 0 + fi + + verify_needed_test mvnsite + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "Determining number of patched site errors" + + personality_modules patch mvnsite + modules_workers patch mvnsite clean site site:stage -Dmaven.javadoc.skip=true + result=$? + modules_messages patch mvnsite true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + + +## @description Make sure Maven's eclipse generation works. +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function mvneclipse_postapply +{ + if [[ ${BUILDTOOL} != maven ]]; then + return 0 + fi + + big_console_header "Verifying mvn eclipse:eclipse still works" + + verify_needed_test javac + if [[ $? == 0 ]]; then + echo "Patch does not touch any java files. Skipping mvn eclipse:eclipse" + return 0 + fi + + personality_modules patch mvneclipse + modules_workers patch mvneclipse eclipse:eclipse + result=$? + modules_messages patch mvneclipse true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + +## @description Verify mvn install works +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function maven_precheck_install +{ + local result=0 + + if [[ ${BUILDTOOL} != maven ]]; then + return 0 + fi + + big_console_header "Verifying mvn install works" + + verify_needed_test javadoc + retval=$? + + verify_needed_test javac + ((retval = retval + $? )) + if [[ ${retval} == 0 ]]; then + echo "This patch does not appear to need mvn install checks." + return 0 + fi + + personality_modules branch mvninstall + modules_workers branch mvninstall -fae clean install -Dmaven.javadoc.skip=true + result=$? + modules_messages branch mvninstall true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} + +## @description Verify mvn install works +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function maven_postapply_install +{ + local result=0 + + if [[ ${BUILDTOOL} != maven ]]; then + return 0 + fi + + big_console_header "Verifying mvn install still works" + + verify_needed_test javadoc + retval=$? + + verify_needed_test javac + ((retval = retval + $? )) + if [[ ${retval} == 0 ]]; then + echo "This patch does not appear to need mvn install checks." + return 0 + fi + + personality_modules patch mvninstall + modules_workers patch mvninstall clean install -Dmaven.javadoc.skip=true + result=$? + modules_messages patch mvninstall true + if [[ ${result} != 0 ]]; then + return 1 + fi + return 0 +} diff --git a/dev-support/test-patch.d/perlcritic.sh b/dev-support/test-patch.d/perlcritic.sh new file mode 100755 index 0000000000000..1cec3f3630373 --- /dev/null +++ b/dev-support/test-patch.d/perlcritic.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_plugin perlcritic + +PERLCRITIC_TIMER=0 + +PERLCRITIC=${PERLCRITIC:-$(which perlcritic 2>/dev/null)} + +function perlcritic_usage +{ + echo "Perl::Critic specific:" + echo "--perlcritic= path to perlcritic executable" +} + +function perlcritic_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --perlcritic=*) + PERLCRITIC=${i#*=} + ;; + esac + done +} + +function perlcritic_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.p[lm]$ ]]; then + add_test perlcritic + fi +} + +function perlcritic_preapply +{ + local i + + verify_needed_test perlcritic + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "Perl::Critic plugin: prepatch" + + if [[ ! -x ${PERLCRITIC} ]]; then + yetus_error "${PERLCRITIC} does not exist." + return 0 + fi + + start_clock + + echo "Running perlcritic against modified perl scripts/modules." + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.p[lm]$ && -f ${i} ]]; then + ${PERLCRITIC} -1 --verbose 1 "${i}" 2>/dev/null >> "${PATCH_DIR}/branch-perlcritic-result.txt" + fi + done + popd >/dev/null + # keep track of how much as elapsed for us already + PERLCRITIC_TIMER=$(stop_clock) + return 0 +} + +function perlcritic_postapply +{ + local i + local numPrepatch + local numPostpatch + local diffPostpatch + + verify_needed_test perlcritic + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "Perl::Critic plugin: postpatch" + + if [[ ! -x ${PERLCRITIC} ]]; then + yetus_error "${PERLCRITIC} is not available." + add_vote_table 0 perlcritic "Perl::Critic was not available." + return 0 + fi + + start_clock + + # add our previous elapsed to our new timer + # by setting the clock back + offset_clock "${PERLCRITIC_TIMER}" + + echo "Running perlcritic against modified perl scripts/modules." + # we re-check this in case one has been added + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.p[lm]$ && -f ${i} ]]; then + ${PERLCRITIC} -1 --verbose 1 "${i}" 2>/dev/null >> "${PATCH_DIR}/patch-perlcritic-result.txt" + fi + done + popd >/dev/null + + PERLCRITIC_VERSION=$(${PERLCRITIC} --version 2>/dev/null) + add_footer_table perlcritic "v${PERLCRITIC_VERSION}" + + calcdiffs "${PATCH_DIR}/branch-perlcritic-result.txt" "${PATCH_DIR}/patch-perlcritic-result.txt" > "${PATCH_DIR}/diff-patch-perlcritic.txt" + # shellcheck disable=SC2016 + diffPostpatch=$(wc -l "${PATCH_DIR}/diff-patch-perlcritic.txt" | ${AWK} '{print $1}') + + if [[ ${diffPostpatch} -gt 0 ]] ; then + # shellcheck disable=SC2016 + numPrepatch=$(wc -l "${PATCH_DIR}/branch-perlcritic-result.txt" | ${AWK} '{print $1}') + + # shellcheck disable=SC2016 + numPostpatch=$(wc -l "${PATCH_DIR}/patch-perlcritic-result.txt" | ${AWK} '{print $1}') + + add_vote_table -1 perlcritic "The applied patch generated "\ + "${diffPostpatch} new Perl::Critic issues (total was ${numPrepatch}, now ${numPostpatch})." + add_footer_table perlcritic "@@BASE@@/diff-patch-perlcritic.txt" + return 1 + fi + + add_vote_table +1 perlcritic "There were no new perlcritic issues." + return 0 +} diff --git a/dev-support/test-patch.d/pylint.sh b/dev-support/test-patch.d/pylint.sh new file mode 100755 index 0000000000000..ebac162156a93 --- /dev/null +++ b/dev-support/test-patch.d/pylint.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_plugin pylint + +PYLINT_TIMER=0 + +PYLINT=${PYLINT:-$(which pylint 2>/dev/null)} +PYLINT_OPTIONS=${PYLINT_OPTIONS:-} + +function pylint_usage +{ + echo "Pylint specific:" + echo "--pylint= path to pylint executable" + echo "--pylint-options= pylint options other than output-format and reports" +} + +function pylint_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --pylint=*) + PYLINT=${i#*=} + ;; + --pylint-options=*) + PYLINT_OPTIONS=${i#*=} + ;; + esac + done +} + +function pylint_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.py$ ]]; then + add_test pylint + fi +} + +function pylint_preapply +{ + local i + + verify_needed_test pylint + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "pylint plugin: prepatch" + + if [[ ! -x ${PYLINT} ]]; then + yetus_error "${PYLINT} does not exist." + return 0 + fi + + start_clock + + echo "Running pylint against modified python scripts." + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.py$ && -f ${i} ]]; then + # shellcheck disable=SC2086 + eval "${PYLINT} ${PYLINT_OPTIONS} --output-format=parseable --reports=n ${i}" 2>/dev/null | + ${AWK} '1> "${PATCH_DIR}/branch-pylint-result.txt" + fi + done + popd >/dev/null + # keep track of how much as elapsed for us already + PYLINT_TIMER=$(stop_clock) + return 0 +} + +function pylint_postapply +{ + local i + local numPrepatch + local numPostpatch + local diffPostpatch + + verify_needed_test pylint + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "pylint plugin: postpatch" + + if [[ ! -x ${PYLINT} ]]; then + yetus_error "${PYLINT} is not available." + add_vote_table 0 pylint "Pylint was not available." + return 0 + fi + + start_clock + + # add our previous elapsed to our new timer + # by setting the clock back + offset_clock "${PYLINT_TIMER}" + + echo "Running pylint against modified python scripts." + # we re-check this in case one has been added + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.py$ && -f ${i} ]]; then + # shellcheck disable=SC2086 + eval "${PYLINT} ${PYLINT_OPTIONS} --output-format=parseable --reports=n ${i}" 2>/dev/null | + ${AWK} '1> "${PATCH_DIR}/patch-pylint-result.txt" + fi + done + popd >/dev/null + + # shellcheck disable=SC2016 + PYLINT_VERSION=$(${PYLINT} --version 2>/dev/null | ${GREP} pylint | ${AWK} '{print $NF}') + add_footer_table pylint "v${PYLINT_VERSION%,}" + + calcdiffs "${PATCH_DIR}/branch-pylint-result.txt" "${PATCH_DIR}/patch-pylint-result.txt" > "${PATCH_DIR}/diff-patch-pylint.txt" + diffPostpatch=$(${AWK} 'BEGIN {sum=0} 2/dev/null)} + +function rubocop_usage +{ + echo "Rubocop specific:" + echo "--rubocop= path to rubocop executable" +} + +function rubocop_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --rubocop=*) + RUBOCOP=${i#*=} + ;; + esac + done +} + +function rubocop_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.rb$ ]]; then + add_test rubocop + fi +} + +function rubocop_preapply +{ + local i + + verify_needed_test rubocop + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "rubocop plugin: prepatch" + + if [[ ! -x ${RUBOCOP} ]]; then + yetus_error "${RUBOCOP} does not exist." + return 0 + fi + + start_clock + + echo "Running rubocop against modified ruby scripts." + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.rb$ && -f ${i} ]]; then + ${RUBOCOP} -f c "${i}" | ${AWK} '!/[0-9]* files? inspected/' >> "${PATCH_DIR}/branch-rubocop-result.txt" + fi + done + popd >/dev/null + # keep track of how much as elapsed for us already + RUBOCOP_TIMER=$(stop_clock) + return 0 +} + +function rubocop_postapply +{ + local i + local numPrepatch + local numPostpatch + local diffPostpatch + + verify_needed_test rubocop + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "rubocop plugin: postpatch" + + if [[ ! -x ${RUBOCOP} ]]; then + yetus_error "${RUBOCOP} is not available." + add_vote_table 0 rubocop "Rubocop was not available." + return 0 + fi + + start_clock + + # add our previous elapsed to our new timer + # by setting the clock back + offset_clock "${RUBOCOP_TIMER}" + + echo "Running rubocop against modified ruby scripts." + # we re-check this in case one has been added + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.rb$ && -f ${i} ]]; then + ${RUBOCOP} -f c "${i}" | ${AWK} '!/[0-9]* files? inspected/' >> "${PATCH_DIR}/patch-rubocop-result.txt" + fi + done + popd >/dev/null + + # shellcheck disable=SC2016 + RUBOCOP_VERSION=$(${RUBOCOP} -v | ${AWK} '{print $NF}') + add_footer_table rubocop "v${RUBOCOP_VERSION}" + + calcdiffs "${PATCH_DIR}/branch-rubocop-result.txt" "${PATCH_DIR}/patch-rubocop-result.txt" > "${PATCH_DIR}/diff-patch-rubocop.txt" + diffPostpatch=$(${AWK} -F: 'BEGIN {sum=0} 4/dev/null)} + +function ruby_lint_usage +{ + echo "Ruby-lint specific:" + echo "--ruby-lint= path to ruby-lint executable" +} + +function ruby_lint_parse_args +{ + local i + + for i in "$@"; do + case ${i} in + --ruby-lint=*) + RUBY_LINT=${i#*=} + ;; + esac + done +} + +function ruby_lint_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.rb$ ]]; then + add_test ruby_lint + fi +} + +function ruby_lint_preapply +{ + local i + + verify_needed_test ruby_lint + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "ruby-lint plugin: prepatch" + + if [[ ! -x ${RUBY_LINT} ]]; then + yetus_error "${RUBY_LINT} does not exist." + return 0 + fi + + start_clock + + echo "Running ruby-lint against modified ruby scripts." + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.rb$ && -f ${i} ]]; then + ${RUBY_LINT} -p syntastic "${i}" | sort -t : -k 1,1 -k 3,3n -k 4,4n >> "${PATCH_DIR}/branch-ruby-lint-result.txt" + fi + done + popd >/dev/null + # keep track of how much as elapsed for us already + RUBY_LINT_TIMER=$(stop_clock) + return 0 +} + +function ruby_lint_postapply +{ + local i + local numPrepatch + local numPostpatch + local diffPostpatch + + verify_needed_test ruby_lint + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "ruby-lint plugin: postpatch" + + if [[ ! -x ${RUBY_LINT} ]]; then + yetus_error "${RUBY_LINT} is not available." + add_vote_table 0 ruby-lint "Ruby-lint was not available." + return 0 + fi + + start_clock + + # add our previous elapsed to our new timer + # by setting the clock back + offset_clock "${RUBY_LINT_TIMER}" + + echo "Running ruby-lint against modified ruby scripts." + # we re-check this in case one has been added + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ \.rb$ && -f ${i} ]]; then + ${RUBY_LINT} -p syntastic "${i}" | sort -t : -k 1,1 -k 3,3n -k 4,4n >> "${PATCH_DIR}/patch-ruby-lint-result.txt" + fi + done + popd >/dev/null + + # shellcheck disable=SC2016 + RUBY_LINT_VERSION=$(${RUBY_LINT} -v | ${AWK} '{print $2}') + add_footer_table ruby-lint "${RUBY_LINT_VERSION}" + + calcdiffs "${PATCH_DIR}/branch-ruby-lint-result.txt" "${PATCH_DIR}/patch-ruby-lint-result.txt" > "${PATCH_DIR}/diff-patch-ruby-lint.txt" + diffPostpatch=$(${AWK} -F: 'BEGIN {sum=0} 4 "${PATCH_DIR}/branch-scaladoc-${fn}-warning.txt" + else + touch "${PATCH_DIR}/branch-scaladoc-${fn}.txt" \ + "${PATCH_DIR}/branch-scaladoc-${fn}-warning.txt" + fi + + if [[ -f "${PATCH_DIR}/patch-scaladoc-${fn}.txt" ]]; then + ${GREP} -i scaladoc "${PATCH_DIR}/patch-scaladoc-${fn}.txt" \ + > "${PATCH_DIR}/patch-scaladoc-${fn}-warning.txt" + else + touch "${PATCH_DIR}/patch-scaladoc-${fn}.txt" \ + "${PATCH_DIR}/patch-scaladoc-${fn}-warning.txt" + fi + + numbranch=$(${BUILDTOOL}_count_scaladoc_probs "${PATCH_DIR}/branch-scaladoc-${fn}.txt") + numpatch=$(${BUILDTOOL}_count_scaladoc_probs "${PATCH_DIR}/patch-scaladoc-${fn}.txt") + + if [[ -n ${numbranch} + && -n ${numpatch} + && ${numpatch} -gt ${numbranch} ]] ; then + + ${DIFF} -u "${PATCH_DIR}/branch-scaladoc-${fn}-warning.txt" \ + "${PATCH_DIR}/patch-scaladoc-${fn}-warning.txt" \ + > "${PATCH_DIR}/scaladoc-${fn}-diff.txt" + + module_status ${i} -1 "scaladoc-${fn}-diff.txt" \ + "Patched ${module_suffix} generated "\ + "$((numpatch-numbranch)) additional warning messages." + ((result=result+1)) + fi + ((i=i+1)) + done + + modules_messages patch scaladoc true + + if [[ ${result} -gt 0 ]]; then + return 1 + fi + return 0 +} diff --git a/dev-support/test-patch.d/shellcheck.sh b/dev-support/test-patch.d/shellcheck.sh index 9277ea50aa475..0c198db19a01b 100755 --- a/dev-support/test-patch.d/shellcheck.sh +++ b/dev-support/test-patch.d/shellcheck.sh @@ -47,8 +47,16 @@ function shellcheck_private_findbash while read line; do value=$(find "${line}" ! -name '*.cmd' -type f \ | ${GREP} -E -v '(.orig$|.rej$)') - list="${list} ${value}" - done < <(find . -type d -name bin -o -type d -name sbin -o -type d -name libexec -o -type d -name shellprofile.d) + + for i in ${value}; do + if [[ ! ${i} =~ \.sh(\.|$) + && ! $(head -n 1 "${i}") =~ ^#! ]]; then + yetus_debug "Shellcheck skipped: ${i}" + continue + fi + list="${list} ${i}" + done + done < <(find . -type d -name bin -o -type d -name sbin -o -type d -name scripts -o -type d -name libexec -o -type d -name shellprofile.d) # shellcheck disable=SC2086 echo ${list} ${SHELLCHECK_SPECIFICFILES} | tr ' ' '\n' | sort -u } @@ -65,20 +73,17 @@ function shellcheck_preapply big_console_header "shellcheck plugin: prepatch" if [[ ! -x "${SHELLCHECK}" ]]; then - hadoop_error "shellcheck is not available." + yetus_error "shellcheck is not available." return 0 fi start_clock - # shellcheck disable=SC2016 - SHELLCHECK_VERSION=$(${SHELLCHECK} --version | ${GREP} version: | ${AWK} '{print $NF}') - echo "Running shellcheck against all identifiable shell scripts" pushd "${BASEDIR}" >/dev/null for i in $(shellcheck_private_findbash); do if [[ -f ${i} ]]; then - ${SHELLCHECK} -f gcc "${i}" >> "${PATCH_DIR}/${PATCH_BRANCH}shellcheck-result.txt" + ${SHELLCHECK} -f gcc "${i}" >> "${PATCH_DIR}/branch-shellcheck-result.txt" fi done popd > /dev/null @@ -87,48 +92,13 @@ function shellcheck_preapply return 0 } -function shellcheck_calcdiffs -{ - local orig=$1 - local new=$2 - local diffout=$3 - local tmp=${PATCH_DIR}/sc.$$.${RANDOM} - local count=0 - local j - - # first, pull out just the errors - # shellcheck disable=SC2016 - ${AWK} -F: '{print $NF}' "${orig}" >> "${tmp}.branch" - - # shellcheck disable=SC2016 - ${AWK} -F: '{print $NF}' "${new}" >> "${tmp}.patch" - - # compare the errors, generating a string of line - # numbers. Sorry portability: GNU diff makes this too easy - ${DIFF} --unchanged-line-format="" \ - --old-line-format="" \ - --new-line-format="%dn " \ - "${tmp}.branch" \ - "${tmp}.patch" > "${tmp}.lined" - - # now, pull out those lines of the raw output - # shellcheck disable=SC2013 - for j in $(cat "${tmp}.lined"); do - # shellcheck disable=SC2086 - head -${j} "${new}" | tail -1 >> "${diffout}" - done - - if [[ -f "${diffout}" ]]; then - # shellcheck disable=SC2016 - count=$(wc -l "${diffout}" | ${AWK} '{print $1}' ) - fi - rm "${tmp}.branch" "${tmp}.patch" "${tmp}.lined" 2>/dev/null - echo "${count}" -} - function shellcheck_postapply { local i + local msg + local numPrepatch + local numPostpatch + local diffPostpatch verify_needed_test shellcheck if [[ $? == 0 ]]; then @@ -138,8 +108,8 @@ function shellcheck_postapply big_console_header "shellcheck plugin: postpatch" if [[ ! -x "${SHELLCHECK}" ]]; then - hadoop_error "shellcheck is not available." - add_jira_table 0 shellcheck "Shellcheck was not available." + yetus_error "shellcheck is not available." + add_vote_table 0 shellcheck "Shellcheck was not available." return 0 fi @@ -152,27 +122,42 @@ function shellcheck_postapply echo "Running shellcheck against all identifiable shell scripts" # we re-check this in case one has been added for i in $(shellcheck_private_findbash); do - ${SHELLCHECK} -f gcc "${i}" >> "${PATCH_DIR}/patchshellcheck-result.txt" + ${SHELLCHECK} -f gcc "${i}" >> "${PATCH_DIR}/patch-shellcheck-result.txt" done + if [[ ! -f "${PATCH_DIR}/branch-shellcheck-result.txt" ]]; then + touch "${PATCH_DIR}/branch-shellcheck-result.txt" + fi + # shellcheck disable=SC2016 - numPrepatch=$(wc -l "${PATCH_DIR}/${PATCH_BRANCH}shellcheck-result.txt" | ${AWK} '{print $1}') - # shellcheck disable=SC2016 - numPostpatch=$(wc -l "${PATCH_DIR}/patchshellcheck-result.txt" | ${AWK} '{print $1}') + SHELLCHECK_VERSION=$(${SHELLCHECK} --version | ${GREP} version: | ${AWK} '{print $NF}') + msg="v${SHELLCHECK_VERSION}" + if [[ ${SHELLCHECK_VERSION} =~ 0.[0-3].[0-5] ]]; then + msg="${msg} (This is an old version that has serious bugs. Consider upgrading.)" + fi + add_footer_table shellcheck "${msg}" - diffPostpatch=$(shellcheck_calcdiffs \ - "${PATCH_DIR}/${PATCH_BRANCH}shellcheck-result.txt" \ - "${PATCH_DIR}/patchshellcheck-result.txt" \ - "${PATCH_DIR}/diffpatchshellcheck.txt" - ) + calcdiffs \ + "${PATCH_DIR}/branch-shellcheck-result.txt" \ + "${PATCH_DIR}/patch-shellcheck-result.txt" \ + > "${PATCH_DIR}/diff-patch-shellcheck.txt" + # shellcheck disable=SC2016 + diffPostpatch=$(wc -l "${PATCH_DIR}/diff-patch-shellcheck.txt" | ${AWK} '{print $1}') if [[ ${diffPostpatch} -gt 0 ]] ; then - add_jira_table -1 shellcheck "The applied patch generated "\ - "${diffPostpatch} new shellcheck (v${SHELLCHECK_VERSION}) issues (total was ${numPrepatch}, now ${numPostpatch})." - add_jira_footer shellcheck "@@BASE@@/diffpatchshellcheck.txt" + # shellcheck disable=SC2016 + numPrepatch=$(wc -l "${PATCH_DIR}/branch-shellcheck-result.txt" | ${AWK} '{print $1}') + + # shellcheck disable=SC2016 + numPostpatch=$(wc -l "${PATCH_DIR}/patch-shellcheck-result.txt" | ${AWK} '{print $1}') + + add_vote_table -1 shellcheck "The applied patch generated "\ + "${diffPostpatch} new shellcheck issues (total was ${numPrepatch}, now ${numPostpatch})." + add_footer_table shellcheck "@@BASE@@/diff-patch-shellcheck.txt" + bugsystem_linecomments "shellcheck" "${PATCH_DIR}/diff-patch-shellcheck.txt" return 1 fi - add_jira_table +1 shellcheck "There were no new shellcheck (v${SHELLCHECK_VERSION}) issues." + add_vote_table +1 shellcheck "There were no new shellcheck issues." return 0 } diff --git a/dev-support/test-patch.d/tap.sh b/dev-support/test-patch.d/tap.sh new file mode 100755 index 0000000000000..c6796a8692b59 --- /dev/null +++ b/dev-support/test-patch.d/tap.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_test_format tap + +TAP_FAILED_TESTS="" +TAP_LOG_DIR="target/tap" + +function tap_process_args +{ + declare i + + for i in "$@"; do + case ${i} in + --tap-log-dir=*) + TAP_LOG_DIR=${i#=*} + ;; + esac + done +} + +function tap_usage +{ + echo "TAP Options:" + echo "--tap-log-dir= Directory relative to the module for tap output (default: \"target/tap\")" +} + +function tap_process_tests +{ + # shellcheck disable=SC2034 + declare module=$1 + # shellcheck disable=SC2034 + declare buildlogfile=$2 + declare filefrag=$3 + declare result=0 + declare module_failed_tests + declare filenames + + filenames=$(find "${TAP_LOG_DIR}" -type f -exec "${GREP}" -l -E "not ok " {} \;) + + if [[ -n "${filenames}" ]]; then + module_failed_tests=$(echo "${filenames}" \ + | sed -e "s,${TAP_LOG_DIR},,g" -e s,^/,,g ) + # shellcheck disable=SC2086 + cat ${filenames} >> "${PATCH_DIR}/patch-${filefrag}.tap" + TAP_LOGS="${TAP_LOGS} @@BASE@@/patch-${filefrag}.tap" + TAP_FAILED_TESTS="${TAP_FAILED_TESTS} ${module_failed_tests}" + ((result=result+1)) + fi + + if [[ ${result} -gt 0 ]]; then + return 1 + fi + return 0 +} + +function tap_finalize_results +{ + declare jdk=$1 + + if [[ -n "${TAP_FAILED_TESTS}" ]] ; then + # shellcheck disable=SC2086 + populate_test_table "${jdk}Failed junit tests" ${TAP_FAILED_TESTS} + TAP_FAILED_TESTS="" + add_footer_table "TAP logs" "${TAP_LOGS}" + fi +} diff --git a/dev-support/test-patch.d/whitespace.sh b/dev-support/test-patch.d/whitespace.sh index 324481ca74d51..e6676e3f7b04d 100755 --- a/dev-support/test-patch.d/whitespace.sh +++ b/dev-support/test-patch.d/whitespace.sh @@ -16,31 +16,72 @@ add_plugin whitespace + +function whitespace_linecomment_reporter +{ + local file=$1 + shift + local comment=$* + local tmpfile="${PATCH_DIR}/wlr.$$.${RANDOM}" + + while read -r line; do + { + printf "%s" $(echo "${line}" | cut -f1-2 -d:) + echo "${comment}" + } >> "${tmpfile}" + done < "${file}" + + bugsystem_linecomments "whitespace:" "${tmpfile}" + rm "${tmpfile}" +} + function whitespace_postapply { local count - local j + local result=0 big_console_header "Checking for whitespace at the end of lines" start_clock pushd "${BASEDIR}" >/dev/null - for j in ${CHANGED_FILES}; do - ${GREP} -nHE '[[:blank:]]$' "./${j}" | ${GREP} -f "${GITDIFFLINES}" >> "${PATCH_DIR}/whitespace.txt" - done + # shellcheck disable=SC2016 + ${AWK} '/\t/ {print $0}' \ + "${GITDIFFCONTENT}" \ + | ${GREP} -v Makefile: >> "${PATCH_DIR}/whitespace-tabs.txt" + + ${GREP} -E '[[:blank:]]$' \ + "${GITDIFFCONTENT}" \ + >> "${PATCH_DIR}/whitespace-eol.txt" # shellcheck disable=SC2016 - count=$(wc -l "${PATCH_DIR}/whitespace.txt" | ${AWK} '{print $1}') + count=$(wc -l "${PATCH_DIR}/whitespace-eol.txt" | ${AWK} '{print $1}') if [[ ${count} -gt 0 ]]; then - add_jira_table -1 whitespace "The patch has ${count}"\ + add_vote_table -1 whitespace "The patch has ${count}"\ " line(s) that end in whitespace. Use git apply --whitespace=fix." - add_jira_footer whitespace "@@BASE@@/whitespace.txt" + + whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-eol.txt" "end of line" + add_footer_table whitespace "@@BASE@@/whitespace-eol.txt" + ((result=result+1)) + fi + + # shellcheck disable=SC2016 + count=$(wc -l "${PATCH_DIR}/whitespace-tabs.txt" | ${AWK} '{print $1}') + + if [[ ${count} -gt 0 ]]; then + add_vote_table -1 whitespace "The patch has ${count}"\ + " line(s) with tabs." + add_footer_table whitespace "@@BASE@@/whitespace-tabs.txt" + whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-tabs.txt" "tabs in line" + ((result=result+1)) + fi + + if [[ ${result} -gt 0 ]]; then popd >/dev/null return 1 fi popd >/dev/null - add_jira_table +1 whitespace "The patch has no lines that end in whitespace." + add_vote_table +1 whitespace "Patch has no whitespace issues." return 0 } diff --git a/dev-support/test-patch.d/xml.sh b/dev-support/test-patch.d/xml.sh new file mode 100755 index 0000000000000..b33c7cdc42434 --- /dev/null +++ b/dev-support/test-patch.d/xml.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_plugin xml + +function xml_filefilter +{ + local filename=$1 + + if [[ ${filename} =~ \.xml$ ]]; then + add_test xml + fi +} + +function xml_postapply +{ + local js + local i + local count + + verify_needed_test xml + if [[ $? == 0 ]]; then + return 0 + fi + + big_console_header "Checking if XML files are well-formed" + + js="${JAVA_HOME}/bin/jrunscript" + if [[ ! -x ${js} ]]; then + yetus_error "${js} does not exist" + return 0 + fi + + start_clock + + pushd "${BASEDIR}" >/dev/null + for i in ${CHANGED_FILES}; do + if [[ ! ${i} =~ \.xml$ ]]; then + continue + fi + ${js} -e "XMLDocument(arguments[0])" "${i}" >> "${PATCH_DIR}/xml.txt" 2>&1 + if [[ $? != 0 ]]; then + ((count=count+1)) + fi + done + + if [[ ${count} -gt 0 ]]; then + add_vote_table -1 xml "The patch has ${count} ill-formed XML file(s)." + add_footer_table xml "@@BASE@@/xml.txt" + popd >/dev/null + return 1 + fi + + popd >/dev/null + add_vote_table +1 xml "The patch has no ill-formed XML file." + return 0 +} diff --git a/dev-support/test-patch.sh b/dev-support/test-patch.sh index a3cdc85b8d636..bd0a8ea063421 100755 --- a/dev-support/test-patch.sh +++ b/dev-support/test-patch.sh @@ -14,13 +14,40 @@ # See the License for the specific language governing permissions and # limitations under the License. +# Make sure that bash version meets the pre-requisite + +if [[ -z "${BASH_VERSINFO}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." + exit 1 +fi + ### BUILD_URL is set by Hudson if it is run by patch process this="${BASH_SOURCE-$0}" BINDIR=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P) -CWD=$(pwd) +STARTINGDIR=$(pwd) USER_PARAMS=("$@") GLOBALTIMER=$(date +"%s") +#shellcheck disable=SC2034 +QATESTMODE=false + +# global arrays +declare -a TP_HEADER +declare -a TP_VOTE_TABLE +declare -a TP_TEST_TABLE +declare -a TP_FOOTER_TABLE +declare -a MODULE_STATUS +declare -a MODULE_STATUS_TIMER +declare -a MODULE_STATUS_MSG +declare -a MODULE_STATUS_LOG +declare -a MODULE + +TP_HEADER_COUNTER=0 +TP_VOTE_COUNTER=0 +TP_TEST_COUNTER=0 +TP_FOOTER_COUNTER=0 ## @description Setup the default global variables ## @audience public @@ -28,15 +55,8 @@ GLOBALTIMER=$(date +"%s") ## @replaceable no function setup_defaults { - if [[ -z "${MAVEN_HOME:-}" ]]; then - MVN=mvn - else - MVN=${MAVEN_HOME}/bin/mvn - fi - # This parameter needs to be kept as an array - MAVEN_ARGS=() - - PROJECT_NAME=hadoop + PROJECT_NAME=yetus + DOCKERFILE="${BINDIR}/test-patch-docker/Dockerfile-startstub" HOW_TO_CONTRIBUTE="https://wiki.apache.org/hadoop/HowToContribute" JENKINS=false BASEDIR=$(pwd) @@ -44,64 +64,56 @@ function setup_defaults USER_PLUGIN_DIR="" LOAD_SYSTEM_PLUGINS=true + ALLOWSUMMARIES=true - FINDBUGS_HOME=${FINDBUGS_HOME:-} - FINDBUGS_WARNINGS_FAIL_PRECHECK=false + DOCKERSUPPORT=false ECLIPSE_HOME=${ECLIPSE_HOME:-} BUILD_NATIVE=${BUILD_NATIVE:-true} PATCH_BRANCH="" - PATCH_BRANCH_DEFAULT="trunk" + PATCH_BRANCH_DEFAULT="master" + + # shellcheck disable=SC2034 CHANGED_MODULES="" + # shellcheck disable=SC2034 + CHANGED_UNFILTERED_MODULES="" + # shellcheck disable=SC2034 + CHANGED_UNION_MODULES="" USER_MODULE_LIST="" OFFLINE=false CHANGED_FILES="" REEXECED=false RESETREPO=false ISSUE="" - ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$' TIMER=$(date +"%s") - PATCHURL="" - OSTYPE=$(uname -s) + BUILDTOOL=maven + TESTFORMATS="" + JDK_TEST_LIST="javac javadoc unit" # Solaris needs POSIX, not SVID case ${OSTYPE} in SunOS) - PS=${PS:-ps} AWK=${AWK:-/usr/xpg4/bin/awk} SED=${SED:-/usr/xpg4/bin/sed} - WGET=${WGET:-wget} + CURL=${CURL:-curl} GIT=${GIT:-git} - EGREP=${EGREP:-/usr/xpg4/bin/egrep} GREP=${GREP:-/usr/xpg4/bin/grep} - PATCH=${PATCH:-patch} + PATCH=${PATCH:-/usr/gnu/bin/patch} DIFF=${DIFF:-/usr/gnu/bin/diff} - JIRACLI=${JIRA:-jira} FILE=${FILE:-file} ;; *) - PS=${PS:-ps} AWK=${AWK:-awk} SED=${SED:-sed} - WGET=${WGET:-wget} + CURL=${CURL:-curl} GIT=${GIT:-git} - EGREP=${EGREP:-egrep} GREP=${GREP:-grep} PATCH=${PATCH:-patch} DIFF=${DIFF:-diff} - JIRACLI=${JIRA:-jira} FILE=${FILE:-file} ;; esac - declare -a JIRA_COMMENT_TABLE - declare -a JIRA_FOOTER_TABLE - declare -a JIRA_HEADER - declare -a JIRA_TEST_TABLE - - JFC=0 - JTC=0 - JTT=0 RESULT=0 } @@ -110,7 +122,7 @@ function setup_defaults ## @stability stable ## @replaceable no ## @param string -function hadoop_error +function yetus_error { echo "$*" 1>&2 } @@ -120,20 +132,51 @@ function hadoop_error ## @stability stable ## @replaceable no ## @param string -function hadoop_debug +function yetus_debug { - if [[ -n "${HADOOP_SHELL_SCRIPT_DEBUG}" ]]; then + if [[ -n "${TP_SHELL_SCRIPT_DEBUG}" ]]; then echo "[$(date) DEBUG]: $*" 1>&2 fi } +## @description Convert the given module name to a file fragment +## @audience public +## @stability stable +## @replaceable no +## @param module +function module_file_fragment +{ + local mod=$1 + if [[ ${mod} == . ]]; then + echo root + else + echo "$1" | tr '/' '_' | tr '\\' '_' + fi +} + +## @description Convert time in seconds to m + s +## @audience public +## @stability stable +## @replaceable no +## @param seconds +function clock_display +{ + local -r elapsed=$1 + + if [[ ${elapsed} -lt 0 ]]; then + echo "N/A" + else + printf "%3sm %02ss" $((elapsed/60)) $((elapsed%60)) + fi +} + ## @description Activate the local timer ## @audience public ## @stability stable ## @replaceable no function start_clock { - hadoop_debug "Start clock" + yetus_debug "Start clock" TIMER=$(date +"%s") } @@ -145,7 +188,7 @@ function stop_clock { local -r stoptime=$(date +"%s") local -r elapsed=$((stoptime-TIMER)) - hadoop_debug "Stop clock" + yetus_debug "Stop clock" echo ${elapsed} } @@ -158,7 +201,7 @@ function stop_global_clock { local -r stoptime=$(date +"%s") local -r elapsed=$((stoptime-GLOBALTIMER)) - hadoop_debug "Stop global clock" + yetus_debug "Stop global clock" echo ${elapsed} } @@ -178,10 +221,11 @@ function offset_clock ## @stability stable ## @replaceable no ## @param string -function add_jira_header +function add_header_line { - JIRA_HEADER[${JHC}]="| $* |" - JHC=$(( JHC+1 )) + # shellcheck disable=SC2034 + TP_HEADER[${TP_HEADER_COUNTER}]="$*" + ((TP_HEADER_COUNTER=TP_HEADER_COUNTER+1 )) } ## @description Add to the output table. If the first parameter is a number @@ -199,88 +243,173 @@ function add_jira_header ## @param subsystem ## @param string ## @return Elapsed time display -function add_jira_table +function add_vote_table { local value=$1 local subsystem=$2 shift 2 - local color - local calctime=0 - + local calctime local -r elapsed=$(stop_clock) - if [[ ${elapsed} -lt 0 ]]; then - calctime="N/A" + yetus_debug "add_vote_table ${value} ${subsystem} ${*}" + + calctime=$(clock_display "${elapsed}") + + if [[ ${value} == "1" ]]; then + value="+1" + fi + + if [[ -z ${value} ]]; then + # shellcheck disable=SC2034 + TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| | ${subsystem} | | ${*:-} |" else - printf -v calctime "%3sm %02ss" $((elapsed/60)) $((elapsed%60)) + # shellcheck disable=SC2034 + TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| ${value} | ${subsystem} | ${calctime} | $* |" fi + ((TP_VOTE_COUNTER=TP_VOTE_COUNTER+1)) +} - echo "" - echo "Elapsed time: ${calctime}" - echo "" +## @description Report the JVM version of the given directory +## @stability stable +## @audience private +## @replaceable yes +## @params directory +## @returns version +function report_jvm_version +{ + #shellcheck disable=SC2016 + "${1}/bin/java" -version 2>&1 | head -1 | ${AWK} '{print $NF}' | tr -d \" +} - case ${value} in - 1|+1) - value="+1" - color="green" - ;; - -1) - color="red" - ;; - 0) - color="blue" - ;; - null) - ;; - esac +## @description Verify if a given test is multijdk +## @audience public +## @stability stable +## @replaceable yes +## @param test +## @return 1 = yes +## @return 0 = no +function verify_multijdk_test +{ + local i=$1 - if [[ -z ${color} ]]; then - JIRA_COMMENT_TABLE[${JTC}]="| | ${subsystem} | | ${*:-} |" - JTC=$(( JTC+1 )) - else - JIRA_COMMENT_TABLE[${JTC}]="| {color:${color}}${value}{color} | ${subsystem} | ${calctime} | $* |" - JTC=$(( JTC+1 )) + if [[ "${JDK_DIR_LIST}" == "${JAVA_HOME}" ]]; then + yetus_debug "MultiJDK not configured." + return 0 + fi + + if [[ ${JDK_TEST_LIST} =~ $i ]]; then + yetus_debug "${i} is in ${JDK_TEST_LIST} and MultiJDK configured." + return 1 fi + return 0 +} + +## @description Absolute path the JDK_DIR_LIST and JAVA_HOME. +## @description if JAVA_HOME is in JDK_DIR_LIST, it is positioned last +## @stability stable +## @audience private +## @replaceable yes +function fullyqualifyjdks +{ + local i + local jdkdir + local tmplist + + JAVA_HOME=$(cd -P -- "${JAVA_HOME}" >/dev/null && pwd -P) + + for i in ${JDK_DIR_LIST}; do + jdkdir=$(cd -P -- "${i}" >/dev/null && pwd -P) + if [[ ${jdkdir} != "${JAVA_HOME}" ]]; then + tmplist="${tmplist} ${jdkdir}" + fi + done + + JDK_DIR_LIST="${tmplist} ${JAVA_HOME}" + JDK_DIR_LIST=${JDK_DIR_LIST/ } } -## @description Put the final environment information at the bottom +## @description Put the opening environment information at the bottom ## @description of the footer table ## @stability stable ## @audience private ## @replaceable yes -function close_jira_footer +function prepopulate_footer { # shellcheck disable=SC2016 - local -r javaversion=$("${JAVA_HOME}/bin/java" -version 2>&1 | head -1 | ${AWK} '{print $NF}' | tr -d \") + local javaversion + local listofjdks local -r unamea=$(uname -a) + local i + + add_footer_table "uname" "${unamea}" + add_footer_table "Build tool" "${BUILDTOOL}" + + if [[ -n ${PERSONALITY} ]]; then + add_footer_table "Personality" "${PERSONALITY}" + fi + + javaversion=$(report_jvm_version "${JAVA_HOME}") + add_footer_table "Default Java" "${javaversion}" + if [[ -n ${JDK_DIR_LIST} + && ${JDK_DIR_LIST} != "${JAVA_HOME}" ]]; then + for i in ${JDK_DIR_LIST}; do + javaversion=$(report_jvm_version "${i}") + listofjdks="${listofjdks} ${i}:${javaversion}" + done + add_footer_table "Multi-JDK versions" "${listofjdks}" + fi +} + +## @description Put docker stats in various tables +## @stability stable +## @audience private +## @replaceable yes +function finish_docker_stats +{ + if [[ ${DOCKERMODE} == true ]]; then + # DOCKER_VERSION is set by our creator. + add_footer_table "Docker" "${DOCKER_VERSION}" + fi +} - add_jira_footer "Java" "${javaversion}" - add_jira_footer "uname" "${unamea}" +## @description Put the max memory consumed by maven at the bottom of the table. +## @audience private +## @stability stable +## @replaceable no +function finish_footer_table +{ + local maxmem + + # `sort | head` can cause a broken pipe error, but we can ignore it just like compute_gitdiff. + # shellcheck disable=SC2016,SC2086 + maxmem=$(find "${PATCH_DIR}" -type f -exec ${AWK} 'match($0, /^\[INFO\] Final Memory: [0-9]+/) + { print substr($0, 22, RLENGTH-21) }' {} \; | sort -nr 2>/dev/null | head -n 1) + + if [[ -n ${maxmem} ]]; then + add_footer_table "Max memory used" "${maxmem}MB" + fi } ## @description Put the final elapsed time at the bottom of the table. ## @audience private ## @stability stable ## @replaceable no -function close_jira_table +function finish_vote_table { local -r elapsed=$(stop_global_clock) + local calctime - if [[ ${elapsed} -lt 0 ]]; then - calctime="N/A" - else - printf -v calctime "%3sm %02ss" $((elapsed/60)) $((elapsed%60)) - fi + calctime=$(clock_display "${elapsed}") echo "" echo "Total Elapsed time: ${calctime}" echo "" - - JIRA_COMMENT_TABLE[${JTC}]="| | | ${calctime} | |" - JTC=$(( JTC+1 )) + # shellcheck disable=SC2034 + TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| | | ${calctime} | |" + ((TP_VOTE_COUNTER=TP_VOTE_COUNTER+1 )) } ## @description Add to the footer of the display. @@BASE@@ will get replaced with the @@ -291,13 +420,14 @@ function close_jira_table ## @replaceable no ## @param subsystem ## @param string -function add_jira_footer +function add_footer_table { local subsystem=$1 shift 1 - JIRA_FOOTER_TABLE[${JFC}]="| ${subsystem} | $* |" - JFC=$(( JFC+1 )) + # shellcheck disable=SC2034 + TP_FOOTER_TABLE[${TP_FOOTER_COUNTER}]="| ${subsystem} | $* |" + ((TP_FOOTER_COUNTER=TP_FOOTER_COUNTER+1 )) } ## @description Special table just for unit test failures @@ -306,13 +436,14 @@ function add_jira_footer ## @replaceable no ## @param failurereason ## @param testlist -function add_jira_test_table +function add_test_table { local failure=$1 shift 1 - JIRA_TEST_TABLE[${JTT}]="| ${failure} | $* |" - JTT=$(( JTT+1 )) + # shellcheck disable=SC2034 + TP_TEST_TABLE[${TP_TEST_COUNTER}]="| ${failure} | $* |" + ((TP_TEST_COUNTER=TP_TEST_COUNTER+1 )) } ## @description Large display for the user console @@ -334,29 +465,6 @@ function big_console_header printf "\n\n" } -## @description Remove {color} tags from a string -## @audience public -## @stability stable -## @replaceable no -## @param string -## @return string -function colorstripper -{ - local string=$1 - shift 1 - - local green="" - local white="" - local red="" - local blue="" - - echo "${string}" | \ - ${SED} -e "s,{color:red},${red},g" \ - -e "s,{color:green},${green},g" \ - -e "s,{color:blue},${blue},g" \ - -e "s,{color},${white},g" -} - ## @description Find the largest size of a column of an array ## @audience private ## @stability evolving @@ -369,11 +477,13 @@ function findlargest local a=("$@") local sizeofa=${#a[@]} local i=0 + local string + local maxlen=0 - until [[ ${i} -gt ${sizeofa} ]]; do + until [[ ${i} -eq ${sizeofa} ]]; do # shellcheck disable=SC2086 string=$( echo ${a[$i]} | cut -f$((column + 1)) -d\| ) - if [[ ${#string} -gt $maxlen ]]; then + if [[ ${#string} -gt ${maxlen} ]]; then maxlen=${#string} fi i=$((i+1)) @@ -391,7 +501,7 @@ function find_java_home { start_clock if [[ -z ${JAVA_HOME:-} ]]; then - case $(uname -s) in + case ${OSTYPE} in Darwin) if [[ -z "${JAVA_HOME}" ]]; then if [[ -x /usr/libexec/java_home ]]; then @@ -409,39 +519,27 @@ function find_java_home if [[ -z ${JAVA_HOME:-} ]]; then echo "JAVA_HOME is not defined." - add_jira_table -1 pre-patch "JAVA_HOME is not defined." + add_vote_table -1 pre-patch "JAVA_HOME is not defined." return 1 fi return 0 } -## @description Write the contents of a file to jenkins +## @description Write the contents of a file to all of the bug systems +## @description (so content should avoid special formatting) ## @params filename ## @stability stable ## @audience public -## @returns ${JIRACLI} exit code -function write_to_jira +function write_comment { local -r commentfile=${1} - shift - - local retval + declare bug - if [[ ${OFFLINE} == false - && ${JENKINS} == true ]]; then - export USER=hudson - # shellcheck disable=SC2086 - ${JIRACLI} --comment "$(cat ${commentfile})" \ - -s https://issues.apache.org/jira \ - -a addcomment -u hadoopqa \ - -p "${JIRA_PASSWD}" \ - --issue "${ISSUE}" - retval=$? - ${JIRACLI} -s https://issues.apache.org/jira \ - -a logout -u hadoopqa \ - -p "${JIRA_PASSWD}" - fi - return ${retval} + for bug in ${BUGSYSTEMS}; do + if declare -f ${bug}_write_comment >/dev/null; then + "${bug}_write_comment" "${commentfile}" + fi + done } ## @description Verify that the patch directory is still in working order @@ -457,48 +555,54 @@ function verify_patchdir_still_exists local extra="" if [[ ! -d ${PATCH_DIR} ]]; then - rm "${commentfile}" 2>/dev/null - - echo "(!) The patch artifact directory has been removed! " > "${commentfile}" - echo "This is a fatal error for test-patch.sh. Aborting. " >> "${commentfile}" - echo - cat ${commentfile} - echo - if [[ ${JENKINS} == true ]]; then - if [[ -n ${NODE_NAME} ]]; then - extra=" (node ${NODE_NAME})" - fi - echo "Jenkins${extra} information at ${BUILD_URL} may provide some hints. " >> "${commentfile}" + rm "${commentfile}" 2>/dev/null - write_to_jira ${commentfile} + echo "(!) The patch artifact directory has been removed! " > "${commentfile}" + echo "This is a fatal error for test-patch.sh. Aborting. " >> "${commentfile}" + echo + cat ${commentfile} + echo + if [[ ${JENKINS} == true ]]; then + if [[ -n ${NODE_NAME} ]]; then + extra=" (node ${NODE_NAME})" fi + echo "Jenkins${extra} information at ${BUILD_URL} may provide some hints. " >> "${commentfile}" - rm "${commentfile}" - cleanup_and_exit ${RESULT} + write_comment ${commentfile} fi + + rm "${commentfile}" + cleanup_and_exit ${RESULT} + fi } -## @description generate a list of all files and line numbers that -## @description that were added/changed in the source repo +## @description generate a list of all files and line numbers in $GITDIFFLINES that +## @description that were added/changed in the source repo. $GITDIFFCONTENT +## @description is same file, but also includes the content of those lines ## @audience private ## @stability stable -## @params filename ## @replaceable no function compute_gitdiff { - local outfile=$1 local file local line local startline local counter local numlines local actual + local content + local outfile="${PATCH_DIR}/computegitdiff.${RANDOM}" pushd "${BASEDIR}" >/dev/null - while read line; do + ${GIT} add --all --intent-to-add + while read -r line; do if [[ ${line} =~ ^\+\+\+ ]]; then file="./"$(echo "${line}" | cut -f2- -d/) continue + elif [[ ${file} =~ gradlew.bat$ + || ${file} =~ gradlew$ + || ${file} =~ gradle/wrapper ]]; then + continue elif [[ ${line} =~ ^@@ ]]; then startline=$(echo "${line}" | cut -f3 -d' ' | cut -f1 -d, | tr -d + ) numlines=$(echo "${line}" | cut -f3 -d' ' | cut -s -f2 -d, ) @@ -511,16 +615,94 @@ function compute_gitdiff numlines=1 fi counter=0 - until [[ ${counter} -gt ${numlines} ]]; do + # it isn't obvious, but on MOST platforms under MOST use cases, + # this is faster than using sed, and definitely faster than using + # awk. + # http://unix.stackexchange.com/questions/47407/cat-line-x-to-line-y-on-a-huge-file + # has a good discussion w/benchmarks + # + # note that if tail is still sending data through the pipe, but head gets enough + # to do what was requested, head will exit, leaving tail with a broken pipe. + # we're going to send stderr to /dev/null and ignore the error since head's + # output is really what we're looking for + tail -n "+${startline}" "${file}" 2>/dev/null | head -n ${numlines} > "${outfile}" + oldifs=${IFS} + IFS='' + while read -r content; do ((actual=counter+startline)) - echo "${file}:${actual}:" >> "${outfile}" + echo "${file}:${actual}:" >> "${GITDIFFLINES}" + printf "%s:%s:%s\n" "${file}" "${actual}" "${content}" >> "${GITDIFFCONTENT}" ((counter=counter+1)) - done + done < "${outfile}" + rm "${outfile}" + IFS=${oldifs} fi done < <("${GIT}" diff --unified=0 --no-color) + + if [[ ! -f "${GITDIFFLINES}" ]]; then + touch "${GITDIFFLINES}" + fi + + if [[ ! -f "${GITDIFFCONTENT}" ]]; then + touch "${GITDIFFCONTENT}" + fi + + if [[ -s "${GITDIFFLINES}" ]]; then + compute_unidiff + else + touch "${GITUNIDIFFLINES}" + fi + popd >/dev/null } +## @description generate an index of unified diff lines vs. modified/added lines +## @description ${GITDIFFLINES} must exist. +## @audience private +## @stability stable +## @replaceable no +function compute_unidiff +{ + declare fn + declare filen + declare tmpfile="${PATCH_DIR}/tmp.$$.${RANDOM}" + + # now that we know what lines are where, we can deal + # with github's pain-in-the-butt API. It requires + # that the client provides the line number of the + # unified diff on a per file basis. + + # First, build a per-file unified diff, pulling + # out the 'extra' lines, grabbing the adds with + # the line number in the diff file along the way, + # finally rewriting the line so that it is in + # './filename:diff line:content' format + + for fn in ${CHANGED_FILES}; do + filen=${fn##./} + + ${GIT} diff ${filen} \ + | tail -n +6 \ + | ${GREP} -n '^+' \ + | ${GREP} -vE '^[0-9]*:\+\+\+' \ + | ${SED} -e 's,^\([0-9]*:\)\+,\1,g' \ + -e s,^,./${filen}:,g \ + >> "${tmpfile}" + done + + # at this point, tmpfile should be in the same format + # as gitdiffcontent, just with different line numbers. + # let's do a merge (using gitdifflines because it's easier) + + # ./filename:real number:diff number + # shellcheck disable=SC2016 + paste -d: "${GITDIFFLINES}" "${tmpfile}" \ + | ${AWK} -F: '{print $1":"$2":"$5":"$6}' \ + >> "${GITUNIDIFFLINES}" + + rm "${tmpfile}" +} + ## @description Print the command to be executing to the screen. Then ## @description run the command, sending stdout and stderr to the given filename ## @description This will also ensure that any directories in ${BASEDIR} have @@ -540,35 +722,41 @@ function echo_and_redirect verify_patchdir_still_exists find "${BASEDIR}" -type d -exec chmod +x {} \; + # to the screen + echo "cd $(pwd)" echo "${*} > ${logfile} 2>&1" - "${@}" > "${logfile}" 2>&1 + # to the log + echo "cd $(pwd)" > "${logfile}" + echo "${*}" >> "${logfile}" + # run the actual command + "${@}" >> "${logfile}" 2>&1 } -## @description is PATCH_DIR relative to BASEDIR? +## @description is a given directory relative to BASEDIR? ## @audience public ## @stability stable ## @replaceable yes -## @returns 1 - no, PATCH_DIR -## @returns 0 - yes, PATCH_DIR - BASEDIR -function relative_patchdir +## @param path +## @returns 1 - no, path +## @returns 0 - yes, path - BASEDIR +function relative_dir { - local p=${PATCH_DIR#${BASEDIR}} + local p=${1#${BASEDIR}} - if [[ ${#p} -eq ${#PATCH_DIR} ]]; then - echo ${p} + if [[ ${#p} -eq ${#1} ]]; then + echo "${p}" return 1 fi p=${p#/} - echo ${p} + echo "${p}" return 0 } - ## @description Print the usage information ## @audience public ## @stability stable ## @replaceable no -function hadoop_usage +function testpatch_usage { local -r up=$(echo ${PROJECT_NAME} | tr '[:lower:]' '[:upper:]') @@ -582,43 +770,57 @@ function hadoop_usage echo "Options:" echo "--basedir= The directory to apply the patch to (default current directory)" echo "--branch= Forcibly set the branch" - echo "--branch-default= If the branch isn't forced and we don't detect one in the patch name, use this branch (default 'trunk')" + echo "--branch-default= If the branch isn't forced and we don't detect one in the patch name, use this branch (default 'master')" echo "--build-native= If true, then build native components (default 'true')" - echo "--contrib-guide= URL to point new users towards project conventions. (default Hadoop's wiki)" + echo "--build-tool= Pick which build tool to focus around (ant, gradle, maven)" + echo "--contrib-guide= URL to point new users towards project conventions. (default: ${HOW_TO_CONTRIBUTE} )" echo "--debug If set, then output some extra stuff to stderr" echo "--dirty-workspace Allow the local git workspace to have uncommitted changes" - echo "--findbugs-home= Findbugs home directory (default FINDBUGS_HOME environment variable)" - echo "--findbugs-strict-precheck If there are Findbugs warnings during precheck, fail" - echo "--issue-re= Bash regular expression to use when trying to find a jira ref in the patch name (default '^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$')" + echo "--docker Spawn a docker container" + echo "--dockerfile= Dockerfile fragment to use as the base" + echo "--java-home= Set JAVA_HOME (In Docker mode, this should be local to the image)" + echo "--multijdkdirs= Comma delimited lists of JDK paths to use for multi-JDK tests" + echo "--multijdktests= Comma delimited tests to use when multijdkdirs is used. (default: javac,javadoc,unit)" echo "--modulelist= Specify additional modules to test (comma delimited)" echo "--offline Avoid connecting to the Internet" - echo "--patch-dir= The directory for working and output files (default '/tmp/${PROJECT_NAME}-test-patch/pid')" + echo "--patch-dir= The directory for working and output files (default '/tmp/test-patch-${PROJECT_NAME}/pid')" + echo "--personality= The personality file to load" echo "--plugins= A directory of user provided plugins. see test-patch.d for examples (default empty)" - echo "--project= The short name for project currently using test-patch (default 'hadoop')" + echo "--project= The short name for project currently using test-patch (default 'yetus')" echo "--resetrepo Forcibly clean the repo" echo "--run-tests Run all relevant tests below the base directory" + echo "--skip-dirs= Skip following directories for module finding" echo "--skip-system-plugins Do not load plugins from ${BINDIR}/test-patch.d" + echo "--summarize= Allow tests to summarize results" echo "--testlist= Specify which subsystem tests to use (comma delimited)" - + echo "--test-parallel= Run multiple tests in parallel (default false in developer mode, true in Jenkins mode)" + echo "--test-threads= Number of tests to run in parallel (default defined in ${PROJECT_NAME} build)" + echo "" echo "Shell binary overrides:" echo "--awk-cmd= The 'awk' command to use (default 'awk')" + echo "--curl-cmd= The 'wget' command to use (default 'curl')" echo "--diff-cmd= The GNU-compatible 'diff' command to use (default 'diff')" echo "--file-cmd= The 'file' command to use (default 'file')" echo "--git-cmd= The 'git' command to use (default 'git')" echo "--grep-cmd= The 'grep' command to use (default 'grep')" - echo "--mvn-cmd= The 'mvn' command to use (default \${MAVEN_HOME}/bin/mvn, or 'mvn')" echo "--patch-cmd= The 'patch' command to use (default 'patch')" - echo "--ps-cmd= The 'ps' command to use (default 'ps')" echo "--sed-cmd= The 'sed' command to use (default 'sed')" echo echo "Jenkins-only options:" echo "--jenkins Run by Jenkins (runs tests and posts results to JIRA)" + echo "--build-url Set the build location web page" echo "--eclipse-home= Eclipse home directory (default ECLIPSE_HOME environment variable)" - echo "--jira-cmd= The 'jira' command to use (default 'jira')" - echo "--jira-password= The password for the 'jira' command" echo "--mv-patch-dir Move the patch-dir into the basedir during cleanup." - echo "--wget-cmd= The 'wget' command to use (default 'wget')" + + importplugins + + for plugin in ${BUILDTOOLS} ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS}; do + if declare -f ${plugin}_usage >/dev/null 2>&1; then + echo + "${plugin}_usage" + fi + done } ## @description Interpret the command line parameters @@ -631,6 +833,7 @@ function parse_args { local i local j + local testlist for i in "$@"; do case ${i} in @@ -649,11 +852,20 @@ function parse_args --build-native=*) BUILD_NATIVE=${i#*=} ;; + --build-tool=*) + BUILDTOOL=${i#*=} + ;; + --build-url=*) + BUILD_URL=${i#*=} + ;; --contrib-guide=*) HOW_TO_CONTRIBUTE=${i#*=} ;; + --curl-cmd=*) + CURL=${i#*=} + ;; --debug) - HADOOP_SHELL_SCRIPT_DEBUG=true + TP_SHELL_SCRIPT_DEBUG=true ;; --diff-cmd=*) DIFF=${i#*=} @@ -661,18 +873,21 @@ function parse_args --dirty-workspace) DIRTY_WORKSPACE=true ;; + --docker) + DOCKERSUPPORT=true + ;; + --dockerfile=*) + DOCKERFILE=${i#*=} + ;; + --dockermode) + DOCKERMODE=true + ;; --eclipse-home=*) ECLIPSE_HOME=${i#*=} ;; --file-cmd=*) FILE=${i#*=} ;; - --findbugs-home=*) - FINDBUGS_HOME=${i#*=} - ;; - --findbugs-strict-precheck) - FINDBUGS_WARNINGS_FAIL_PRECHECK=true - ;; --git-cmd=*) GIT=${i#*=} ;; @@ -680,31 +895,30 @@ function parse_args GREP=${i#*=} ;; --help|-help|-h|help|--h|--\?|-\?|\?) - hadoop_usage + testpatch_usage exit 0 ;; - --issue-re=*) - ISSUE_RE=${i#*=} - ;; --java-home=*) JAVA_HOME=${i#*=} ;; --jenkins) JENKINS=true - ;; - --jira-cmd=*) - JIRACLI=${i#*=} - ;; - --jira-password=*) - JIRA_PASSWD=${i#*=} + TEST_PARALLEL=${TEST_PARALLEL:-true} ;; --modulelist=*) USER_MODULE_LIST=${i#*=} USER_MODULE_LIST=${USER_MODULE_LIST//,/ } - hadoop_debug "Manually forcing modules ${USER_MODULE_LIST}" + yetus_debug "Manually forcing modules ${USER_MODULE_LIST}" + ;; + --multijdkdirs=*) + JDK_DIR_LIST=${i#*=} + JDK_DIR_LIST=${JDK_DIR_LIST//,/ } + yetus_debug "Multi-JVM mode activated with ${JDK_DIR_LIST}" ;; - --mvn-cmd=*) - MVN=${i#*=} + --multijdktests=*) + JDK_TEST_LIST=${i#*=} + JDK_TEST_LIST=${JDK_TEST_LIST//,/ } + yetus_debug "Multi-JVM test list: ${JDK_TEST_LIST}" ;; --mv-patch-dir) RELOCATE_PATCH_DIR=true; @@ -718,19 +932,17 @@ function parse_args --patch-dir=*) USER_PATCH_DIR=${i#*=} ;; + --personality=*) + PERSONALITY=${i#*=} + ;; --plugins=*) USER_PLUGIN_DIR=${i#*=} ;; --project=*) PROJECT_NAME=${i#*=} ;; - --ps-cmd=*) - PS=${i#*=} - ;; --reexec) REEXECED=true - start_clock - add_jira_table 0 reexec "dev-support patch detected." ;; --resetrepo) RESETREPO=true @@ -738,19 +950,42 @@ function parse_args --run-tests) RUN_TESTS=true ;; + --skip-dirs=*) + MODULE_SKIPDIRS=${i#*=} + MODULE_SKIPDIRS=${MODULE_SKIPDIRS//,/ } + yetus_debug "Setting skipdirs to ${MODULE_SKIPDIRS}" + ;; --skip-system-plugins) LOAD_SYSTEM_PLUGINS=false ;; + --summarize=*) + ALLOWSUMMARIES=${i#*=} + ;; --testlist=*) testlist=${i#*=} testlist=${testlist//,/ } for j in ${testlist}; do - hadoop_debug "Manually adding patch test subsystem ${j}" + yetus_debug "Manually adding patch test subsystem ${j}" add_test "${j}" done ;; - --wget-cmd=*) - WGET=${i#*=} + --test-parallel=*) + TEST_PARALLEL=${i#*=} + ;; + --test-threads=*) + # shellcheck disable=SC2034 + TEST_THREADS=${i#*=} + ;; + --tpglobaltimer=*) + GLOBALTIMER=${i#*=} + ;; + --tpreexectimer=*) + REEXECLAUNCHTIMER=${i#*=} + ;; + --*) + ## PATCH_OR_ISSUE can't be a --. So this is probably + ## a plugin thing. + continue ;; *) PATCH_OR_ISSUE=${i} @@ -758,45 +993,37 @@ function parse_args esac done - # if we requested offline, pass that to mvn - if [[ ${OFFLINE} == "true" ]] ; then - MAVEN_ARGS=(${MAVEN_ARGS[@]} --offline) + if [[ -n ${REEXECLAUNCHTIMER} ]]; then + TIMER=${REEXECLAUNCHTIMER}; + else + start_clock fi - # we need absolute dir for ${BASEDIR} - cd "${CWD}" - BASEDIR=$(cd -P -- "${BASEDIR}" >/dev/null && pwd -P) - - if [[ ${BUILD_NATIVE} == "true" ]] ; then - NATIVE_PROFILE=-Pnative - REQUIRE_TEST_LIB_HADOOP=-Drequire.test.libhadoop + if [[ ${REEXECED} == true + && ${DOCKERMODE} == true ]]; then + add_vote_table 0 reexec "docker + precommit patch detected." + elif [[ ${REEXECED} == true ]]; then + add_vote_table 0 reexec "precommit patch detected." + elif [[ ${DOCKERMODE} == true ]]; then + add_vote_table 0 reexec "docker mode." fi + if [[ -z "${PATCH_OR_ISSUE}" ]]; then - hadoop_usage + testpatch_usage exit 1 fi - if [[ ${JENKINS} == "true" ]] ; then - echo "Running in Jenkins mode" - ISSUE=${PATCH_OR_ISSUE} - RESETREPO=true - # shellcheck disable=SC2034 - ECLIPSE_PROPERTY="-Declipse.home=${ECLIPSE_HOME}" - else - if [[ ${RESETREPO} == "true" ]] ; then - echo "Running in destructive (--resetrepo) developer mode" - else - echo "Running in developer mode" - fi - JENKINS=false - fi + + # we need absolute dir for ${BASEDIR} + cd "${STARTINGDIR}" + BASEDIR=$(cd -P -- "${BASEDIR}" >/dev/null && pwd -P) if [[ -n ${USER_PATCH_DIR} ]]; then PATCH_DIR="${USER_PATCH_DIR}" else - PATCH_DIR=/tmp/${PROJECT_NAME}-test-patch/$$ + PATCH_DIR=/tmp/test-patch-${PROJECT_NAME}/$$ fi - cd "${CWD}" + cd "${STARTINGDIR}" if [[ ! -d ${PATCH_DIR} ]]; then mkdir -p "${PATCH_DIR}" if [[ $? == 0 ]] ; then @@ -810,27 +1037,53 @@ function parse_args # we need absolute dir for PATCH_DIR PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P) - GITDIFFLINES=${PATCH_DIR}/gitdifflines.txt + if [[ ${JENKINS} == "true" ]]; then + echo "Running in Jenkins mode" + ISSUE=${PATCH_OR_ISSUE} + RESETREPO=true + # shellcheck disable=SC2034 + ECLIPSE_PROPERTY="-Declipse.home=${ECLIPSE_HOME}" + else + if [[ ${RESETREPO} == "true" ]] ; then + echo "Running in destructive (--resetrepo) developer mode" + else + echo "Running in developer mode" + fi + JENKINS=false + fi + + if [[ -n "${USER_PLUGIN_DIR}" ]]; then + USER_PLUGIN_DIR=$(cd -P -- "${USER_PLUGIN_DIR}" >/dev/null && pwd -P) + fi + + GITDIFFLINES="${PATCH_DIR}/gitdifflines.txt" + GITDIFFCONTENT="${PATCH_DIR}/gitdiffcontent.txt" + GITUNIDIFFLINES="${PATCH_DIR}/gitdiffunilines.txt" + } -## @description Locate the pom.xml file for a given directory +## @description Locate the build file for a given directory ## @audience private ## @stability stable ## @replaceable no -## @return directory containing the pom.xml -function find_pom_dir +## @return directory containing the buildfile. Nothing returned if not found. +## @params buildfile +## @params directory +function find_buildfile_dir { - local dir + local buildfile=$1 + local dir=$2 - dir=$(dirname "$1") - - hadoop_debug "Find pom dir for: ${dir}" + yetus_debug "Find ${buildfile} dir for: ${dir}" while builtin true; do - if [[ -f "${dir}/pom.xml" ]];then + if [[ -f "${dir}/${buildfile}" ]];then echo "${dir}" - hadoop_debug "Found: ${dir}" - return + yetus_debug "Found: ${dir}" + return 0 + elif [[ ${dir} == "." ]]; then + yetus_debug "ERROR: ${buildfile} is not found." + return 1 else dir=$(dirname "${dir}") fi @@ -846,51 +1099,159 @@ function find_changed_files { # get a list of all of the files that have been changed, # except for /dev/null (which would be present for new files). - # Additionally, remove any a/ b/ patterns at the front - # of the patch filenames and any revision info at the end + # Additionally, remove any a/ b/ patterns at the front of the patch filenames. # shellcheck disable=SC2016 - CHANGED_FILES=$(${GREP} -E '^(\+\+\+|---) ' "${PATCH_DIR}/patch" \ - | ${SED} \ - -e 's,^....,,' \ - -e 's,^[ab]/,,' \ - | ${GREP} -v /dev/null \ - | ${AWK} '{print $1}' \ - | sort -u) + CHANGED_FILES=$(${AWK} 'function p(s){sub("^[ab]/","",s); if(s!~"^/dev/null"){print s}} + /^diff --git / { p($3); p($4) } + /^(\+\+\+|---) / { p($2) }' "${PATCH_DIR}/patch" | sort -u) } -## @description Find the modules of the maven build that ${PATCH_DIR}/patch modifies -## @audience private -## @stability stable -## @replaceable no -## @return None; sets ${CHANGED_MODULES} -function find_changed_modules +## @description Check for directories to skip during +## @description changed module calcuation +## @audience private +## @stability stable +## @replaceable no +## @params directory +## @returns 0 for use +## @returns 1 for skip +function module_skipdir { - # Come up with a list of changed files into ${TMP} - local pomdirs - local module - local pommods + local dir=${1} + local i - # Now find all the modules that were changed - for file in ${CHANGED_FILES}; do - #shellcheck disable=SC2086 - pomdirs="${pomdirs} $(find_pom_dir ${file})" - done + yetus_debug "Checking skipdirs for ${dir}" + + if [[ -z ${MODULE_SKIPDIRS} ]]; then + yetus_debug "Skipping skipdirs" + return 0 + fi - # Filter out modules without code - for module in ${pomdirs}; do - ${GREP} "pom" "${module}/pom.xml" > /dev/null - if [[ "$?" != 0 ]]; then - pommods="${pommods} ${module}" + while builtin true; do + for i in ${MODULE_SKIPDIRS}; do + if [[ ${dir} = "${i}" ]];then + yetus_debug "Found a skip: ${dir}" + return 1 + fi + done + if [[ ${dir} == "." ]]; then + return 0 + else + dir=$(dirname "${dir}") + yetus_debug "Trying to skip: ${dir}" fi done - - #shellcheck disable=SC2086 - CHANGED_MODULES=$(echo ${pommods} ${USER_MODULE_LIST} | tr ' ' '\n' | sort -u) } -## @description git checkout the appropriate branch to test. Additionally, this calls -## @description 'determine_issue' and 'determine_branch' based upon the context provided -## @description in ${PATCH_DIR} and in git after checkout. +## @description Find the modules of the build that ${PATCH_DIR}/patch modifies +## @audience private +## @stability stable +## @replaceable no +## @return None; sets ${CHANGED_MODULES} and ${CHANGED_UNFILTERED_MODULES} +function find_changed_modules +{ + local i + local changed_dirs + local builddirs + local builddir + local module + local buildmods + local prev_builddir + local i=1 + local dir + local buildfile + + buildfile=$(${BUILDTOOL}_buildfile) + + if [[ $? != 0 ]]; then + yetus_error "ERROR: Unsupported build tool." + bugsystem_finalreport 1 + cleanup_and_exit 1 + fi + + changed_dirs=$(for i in ${CHANGED_FILES}; do dirname "${i}"; done | sort -u) + + # Now find all the modules that were changed + for i in ${changed_dirs}; do + + module_skipdir "${i}" + if [[ $? != 0 ]]; then + continue + fi + + builddir=$(find_buildfile_dir ${buildfile} "${i}") + if [[ -z ${builddir} ]]; then + yetus_error "ERROR: ${buildfile} is not found. Make sure the target is a ${BUILDTOOL}-based project." + bugsystem_finalreport 1 + cleanup_and_exit 1 + fi + builddirs="${builddirs} ${builddir}" + done + + #shellcheck disable=SC2086,SC2034 + CHANGED_UNFILTERED_MODULES=$(echo ${builddirs} ${USER_MODULE_LIST} | tr ' ' '\n' | sort -u) + #shellcheck disable=SC2086,SC2116 + CHANGED_UNFILTERED_MODULES=$(echo ${CHANGED_UNFILTERED_MODULES}) + + + if [[ ${BUILDTOOL} = maven ]]; then + # Filter out modules without code + for module in ${builddirs}; do + ${GREP} "pom" "${module}/pom.xml" > /dev/null + if [[ "$?" != 0 ]]; then + buildmods="${buildmods} ${module}" + fi + done + else + buildmods=${CHANGED_UNFILTERED_MODULES} + fi + + #shellcheck disable=SC2086,SC2034 + CHANGED_MODULES=$(echo ${buildmods} ${USER_MODULE_LIST} | tr ' ' '\n' | sort -u) + + # turn it back into a list so that anyone printing doesn't + # generate multiline output + #shellcheck disable=SC2086,SC2116 + CHANGED_MODULES=$(echo ${CHANGED_MODULES}) + + yetus_debug "Locate the union of ${CHANGED_MODULES}" + # shellcheck disable=SC2086 + count=$(echo ${CHANGED_MODULES} | wc -w) + if [[ ${count} -lt 2 ]]; then + yetus_debug "Only one entry, so keeping it ${CHANGED_MODULES}" + # shellcheck disable=SC2034 + CHANGED_UNION_MODULES=${CHANGED_MODULES} + return + fi + + i=1 + while [[ ${i} -lt 100 ]] + do + module=$(echo "${CHANGED_MODULES}" | tr ' ' '\n' | cut -f1-${i} -d/ | uniq) + count=$(echo "${module}" | wc -w) + if [[ ${count} -eq 1 + && -f ${module}/${buildfile} ]]; then + prev_builddir=${module} + elif [[ ${count} -gt 1 ]]; then + builddir=${prev_builddir} + break + fi + ((i=i+1)) + done + + if [[ -z ${builddir} ]]; then + builddir="." + fi + + yetus_debug "Finding union of ${builddir}" + builddir=$(find_buildfile_dir ${buildfile} "${builddir}" || true) + + #shellcheck disable=SC2034 + CHANGED_UNION_MODULES="${builddir}" +} + +## @description git checkout the appropriate branch to test. Additionally, this calls +## @description 'determine_issue' and 'determine_branch' based upon the context provided +## @description in ${PATCH_DIR} and in git after checkout. ## @audience private ## @stability stable ## @replaceable no @@ -899,70 +1260,68 @@ function git_checkout { local currentbranch local exemptdir + local status big_console_header "Confirming git environment" cd "${BASEDIR}" if [[ ! -d .git ]]; then - hadoop_error "ERROR: ${BASEDIR} is not a git repo." + yetus_error "ERROR: ${BASEDIR} is not a git repo." cleanup_and_exit 1 fi if [[ ${RESETREPO} == "true" ]] ; then ${GIT} reset --hard if [[ $? != 0 ]]; then - hadoop_error "ERROR: git reset is failing" + yetus_error "ERROR: git reset is failing" cleanup_and_exit 1 fi # if PATCH_DIR is in BASEDIR, then we don't want # git wiping it out. - exemptdir=$(relative_patchdir) + exemptdir=$(relative_dir "${PATCH_DIR}") if [[ $? == 1 ]]; then ${GIT} clean -xdf else # we do, however, want it emptied of all _files_. # we need to leave _directories_ in case we are in # re-exec mode (which places a directory full of stuff in it) - hadoop_debug "Exempting ${exemptdir} from clean" + yetus_debug "Exempting ${exemptdir} from clean" rm "${PATCH_DIR}/*" 2>/dev/null ${GIT} clean -xdf -e "${exemptdir}" fi if [[ $? != 0 ]]; then - hadoop_error "ERROR: git clean is failing" + yetus_error "ERROR: git clean is failing" cleanup_and_exit 1 fi ${GIT} checkout --force "${PATCH_BRANCH_DEFAULT}" if [[ $? != 0 ]]; then - hadoop_error "ERROR: git checkout --force ${PATCH_BRANCH_DEFAULT} is failing" + yetus_error "ERROR: git checkout --force ${PATCH_BRANCH_DEFAULT} is failing" cleanup_and_exit 1 fi determine_branch - if [[ ${PATCH_BRANCH} =~ ^git ]]; then - PATCH_BRANCH=$(echo "${PATCH_BRANCH}" | cut -dt -f2) - fi # we need to explicitly fetch in case the # git ref hasn't been brought in tree yet if [[ ${OFFLINE} == false ]]; then if [[ -f .git/rebase-apply ]]; then - hadoop_error "ERROR: previous rebase failed. Aborting it." + yetus_error "ERROR: previous rebase failed. Aborting it." ${GIT} rebase --abort fi ${GIT} pull --rebase if [[ $? != 0 ]]; then - hadoop_error "ERROR: git pull is failing" + yetus_error "ERROR: git pull is failing" cleanup_and_exit 1 fi fi # forcibly checkout this branch or git ref ${GIT} checkout --force "${PATCH_BRANCH}" if [[ $? != 0 ]]; then - hadoop_error "ERROR: git checkout ${PATCH_BRANCH} is failing" + yetus_error "ERROR: git checkout ${PATCH_BRANCH} is failing" cleanup_and_exit 1 fi @@ -971,7 +1330,7 @@ function git_checkout if [[ ${OFFLINE} == false ]]; then ${GIT} pull --rebase if [[ $? != 0 ]]; then - hadoop_error "ERROR: git pull is failing" + yetus_error "ERROR: git pull is failing" cleanup_and_exit 1 fi fi @@ -980,16 +1339,13 @@ function git_checkout status=$(${GIT} status --porcelain) if [[ "${status}" != "" && -z ${DIRTY_WORKSPACE} ]] ; then - hadoop_error "ERROR: --dirty-workspace option not provided." - hadoop_error "ERROR: can't run in a workspace that contains the following modifications" - hadoop_error "${status}" + yetus_error "ERROR: --dirty-workspace option not provided." + yetus_error "ERROR: can't run in a workspace that contains the following modifications" + yetus_error "${status}" cleanup_and_exit 1 fi determine_branch - if [[ ${PATCH_BRANCH} =~ ^git ]]; then - PATCH_BRANCH=$(echo "${PATCH_BRANCH}" | cut -dt -f2) - fi currentbranch=$(${GIT} rev-parse --abbrev-ref HEAD) if [[ "${currentbranch}" != "${PATCH_BRANCH}" ]];then @@ -1002,8 +1358,6 @@ function git_checkout determine_issue GIT_REVISION=$(${GIT} rev-parse --verify --short HEAD) - # shellcheck disable=SC2034 - VERSION=${GIT_REVISION}_${ISSUE}_PATCH-${patchNum} if [[ "${ISSUE}" == 'Unknown' ]]; then echo "Testing patch on ${PATCH_BRANCH}." @@ -1011,97 +1365,24 @@ function git_checkout echo "Testing ${ISSUE} patch on ${PATCH_BRANCH}." fi - add_jira_footer "git revision" "${PATCH_BRANCH} / ${GIT_REVISION}" - - if [[ ! -f ${BASEDIR}/pom.xml ]]; then - hadoop_error "ERROR: This verison of test-patch.sh only supports Maven-based builds. Aborting." - add_jira_table -1 pre-patch "Unsupported build system." - output_to_jira 1 - cleanup_and_exit 1 - fi - return 0 -} - -## @description Confirm the source environment is compilable -## @audience private -## @stability stable -## @replaceable no -## @return 0 on success -## @return 1 on failure -function precheck_without_patch -{ - local -r mypwd=$(pwd) - - big_console_header "Pre-patch ${PATCH_BRANCH} Java verification" - - start_clock - - verify_needed_test javac - - if [[ $? == 1 ]]; then - echo "Compiling ${mypwd}" - echo_and_redirect "${PATCH_DIR}/${PATCH_BRANCH}JavacWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean test -DskipTests -D${PROJECT_NAME}PatchProcess -Ptest-patch - if [[ $? != 0 ]] ; then - echo "${PATCH_BRANCH} compilation is broken?" - add_jira_table -1 pre-patch "${PATCH_BRANCH} compilation may be broken." - return 1 - fi - else - echo "Patch does not appear to need javac tests." - fi - - verify_needed_test javadoc - - if [[ $? == 1 ]]; then - echo "Javadoc'ing ${mypwd}" - echo_and_redirect "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean test javadoc:javadoc -DskipTests -Pdocs -D${PROJECT_NAME}PatchProcess - if [[ $? != 0 ]] ; then - echo "Pre-patch ${PATCH_BRANCH} javadoc compilation is broken?" - add_jira_table -1 pre-patch "Pre-patch ${PATCH_BRANCH} JavaDoc compilation may be broken." - return 1 - fi - else - echo "Patch does not appear to need javadoc tests." - fi - - verify_needed_test site - - if [[ $? == 1 ]]; then - echo "site creation for ${mypwd}" - echo_and_redirect "${PATCH_DIR}/${PATCH_BRANCH}SiteWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean site site:stage -DskipTests -Dmaven.javadoc.skip=true -D${PROJECT_NAME}PatchProcess - if [[ $? != 0 ]] ; then - echo "Pre-patch ${PATCH_BRANCH} site compilation is broken?" - add_jira_table -1 pre-patch "Pre-patch ${PATCH_BRANCH} site compilation may be broken." - return 1 - fi - else - echo "Patch does not appear to need site tests." - fi - - precheck_findbugs - - if [[ $? != 0 ]] ; then - return 1 - fi + add_footer_table "git revision" "${PATCH_BRANCH} / ${GIT_REVISION}" - add_jira_table 0 pre-patch "Pre-patch ${PATCH_BRANCH} compilation is healthy." return 0 } -## @description Confirm the given branch is a member of the list of space -## @description delimited branches or a git ref +## @description Confirm the given branch is a git reference +## @descriptoin or a valid gitXYZ commit hash ## @audience private ## @stability evolving ## @replaceable no ## @param branch -## @param branchlist -## @return 0 on success +## @return 0 on success, if gitXYZ was passed, PATCH_BRANCH=xyz ## @return 1 on failure function verify_valid_branch { - local branches=$1 - local check=$2 + local check=$1 local i + local hash # shortcut some common # non-resolvable names @@ -1109,26 +1390,22 @@ function verify_valid_branch return 1 fi - if [[ ${check} == patch ]]; then - return 1 - fi - if [[ ${check} =~ ^git ]]; then - ref=$(echo "${check}" | cut -f2 -dt) - count=$(echo "${ref}" | wc -c | tr -d ' ') - - if [[ ${count} == 8 || ${count} == 41 ]]; then - return 0 + hash=$(echo "${check}" | cut -f2- -dt) + if [[ -n ${hash} ]]; then + ${GIT} cat-file -t "${hash}" >/dev/null 2>&1 + if [[ $? -eq 0 ]]; then + PATCH_BRANCH=${hash} + return 0 + fi + return 1 + else + return 1 fi - return 1 fi - for i in ${branches}; do - if [[ "${i}" == "${check}" ]]; then - return 0 - fi - done - return 1 + ${GIT} show-ref "${check}" >/dev/null 2>&1 + return $? } ## @description Try to guess the branch being tested using a variety of heuristics @@ -1139,10 +1416,8 @@ function verify_valid_branch ## @return 1 on failure, with PATCH_BRANCH updated to PATCH_BRANCH_DEFAULT function determine_branch { - local allbranches - local patchnamechunk - - hadoop_debug "Determine branch" + declare bugs + declare retval=1 # something has already set this, so move on if [[ -n ${PATCH_BRANCH} ]]; then @@ -1151,6 +1426,13 @@ function determine_branch pushd "${BASEDIR}" > /dev/null + yetus_debug "Determine branch" + + # something has already set this, so move on + if [[ -n ${PATCH_BRANCH} ]]; then + return + fi + # developer mode, existing checkout, whatever if [[ "${DIRTY_WORKSPACE}" == true ]];then PATCH_BRANCH=$(${GIT} rev-parse --abbrev-ref HEAD) @@ -1158,50 +1440,19 @@ function determine_branch return fi - allbranches=$(${GIT} branch -r | tr -d ' ' | ${SED} -e s,origin/,,g) - - for j in "${PATCHURL}" "${PATCH_OR_ISSUE}"; do - hadoop_debug "Determine branch: starting with ${j}" - # shellcheck disable=SC2016 - patchnamechunk=$(echo "${j}" | ${AWK} -F/ '{print $NF}') - - # ISSUE.branch.##.patch - hadoop_debug "Determine branch: ISSUE.branch.##.patch" - PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2 -d. ) - verify_valid_branch "${allbranches}" "${PATCH_BRANCH}" - if [[ $? == 0 ]]; then - return - fi - - # ISSUE-branch-##.patch - hadoop_debug "Determine branch: ISSUE-branch-##.patch" - PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1,2 -d-) - verify_valid_branch "${allbranches}" "${PATCH_BRANCH}" - if [[ $? == 0 ]]; then - return - fi - - # ISSUE-##.patch.branch - hadoop_debug "Determine branch: ISSUE-##.patch.branch" - # shellcheck disable=SC2016 - PATCH_BRANCH=$(echo "${patchnamechunk}" | ${AWK} -F. '{print $NF}') - verify_valid_branch "${allbranches}" "${PATCH_BRANCH}" - if [[ $? == 0 ]]; then - return - fi - - # ISSUE-branch.##.patch - hadoop_debug "Determine branch: ISSUE-branch.##.patch" - # shellcheck disable=SC2016 - PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | ${AWK} -F. '{print $(NF-2)}' 2>/dev/null) - verify_valid_branch "${allbranches}" "${PATCH_BRANCH}" - if [[ $? == 0 ]]; then - return + for bugs in ${BUGSYSTEMS}; do + if declare -f ${bugs}_determine_branch >/dev/null;then + "${bugs}_determine_branch" + retval=$? + if [[ ${retval} == 0 ]]; then + break + fi fi done - PATCH_BRANCH="${PATCH_BRANCH_DEFAULT}" - + if [[ ${retval} != 0 ]]; then + PATCH_BRANCH="${PATCH_BRANCH_DEFAULT}" + fi popd >/dev/null } @@ -1213,28 +1464,19 @@ function determine_branch ## @return 1 on failure, with ISSUE updated to "Unknown" function determine_issue { - local patchnamechunk - local maybeissue - - hadoop_debug "Determine issue" - - # we can shortcut jenkins - if [[ ${JENKINS} == true ]]; then - ISSUE=${PATCH_OR_ISSUE} - return 0 - fi - - # shellcheck disable=SC2016 - patchnamechunk=$(echo "${PATCH_OR_ISSUE}" | ${AWK} -F/ '{print $NF}') - - maybeissue=$(echo "${patchnamechunk}" | cut -f1,2 -d-) + local bugsys - if [[ ${maybeissue} =~ ${ISSUE_RE} ]]; then - ISSUE=${maybeissue} - return 0 - fi + yetus_debug "Determine issue" - ISSUE="Unknown" + for bugsys in ${BUGSYSTEMS}; do + if declare -f ${bugsys}_determine_issue >/dev/null; then + "${bugsys}_determine_issue" "${PATCH_OR_ISSUE}" + if [[ $? == 0 ]]; then + yetus_debug "${bugsys} says ${ISSUE}" + return 0 + fi + fi + done return 1 } @@ -1247,13 +1489,13 @@ function add_test { local testname=$1 - hadoop_debug "Testing against ${testname}" + yetus_debug "Testing against ${testname}" if [[ -z ${NEEDED_TESTS} ]]; then - hadoop_debug "Setting tests to ${testname}" + yetus_debug "Setting tests to ${testname}" NEEDED_TESTS=${testname} elif [[ ! ${NEEDED_TESTS} =~ ${testname} ]] ; then - hadoop_debug "Adding ${testname}" + yetus_debug "Adding ${testname}" NEEDED_TESTS="${NEEDED_TESTS} ${testname}" fi } @@ -1283,46 +1525,11 @@ function verify_needed_test function determine_needed_tests { local i + local plugin for i in ${CHANGED_FILES}; do - if [[ ${i} =~ src/main/webapp ]]; then - hadoop_debug "tests/webapp: ${i}" - elif [[ ${i} =~ \.sh - || ${i} =~ \.cmd - ]]; then - hadoop_debug "tests/shell: ${i}" - elif [[ ${i} =~ \.md$ - || ${i} =~ \.md\.vm$ - || ${i} =~ src/site - || ${i} =~ src/main/docs - ]]; then - hadoop_debug "tests/site: ${i}" - add_test site - elif [[ ${i} =~ \.c$ - || ${i} =~ \.cc$ - || ${i} =~ \.h$ - || ${i} =~ \.hh$ - || ${i} =~ \.proto$ - || ${i} =~ src/test - || ${i} =~ \.cmake$ - || ${i} =~ CMakeLists.txt - ]]; then - hadoop_debug "tests/units: ${i}" - add_test javac - add_test unit - elif [[ ${i} =~ pom.xml$ - || ${i} =~ \.java$ - || ${i} =~ src/main - ]]; then - hadoop_debug "tests/javadoc+units: ${i}" - add_test javadoc - add_test javac - add_test unit - fi - - if [[ ${i} =~ \.java$ ]]; then - add_test findbugs - fi + yetus_debug "Determining needed tests for ${i}" + personality_file_tests "${i}" for plugin in ${PLUGINS}; do if declare -f ${plugin}_filefilter >/dev/null 2>&1; then @@ -1331,7 +1538,7 @@ function determine_needed_tests done done - add_jira_footer "Optional Tests" "${NEEDED_TESTS}" + add_footer_table "Optional Tests" "${NEEDED_TESTS}" } ## @description Given ${PATCH_ISSUE}, determine what type of patch file is in use, and do the @@ -1343,68 +1550,51 @@ function determine_needed_tests ## @return 1 on failure, may exit function locate_patch { - local notSureIfPatch=false - hadoop_debug "locate patch" + local bugsys + local patchfile="" + local gotit=false + + yetus_debug "locate patch" + # it's a locally provided file if [[ -f ${PATCH_OR_ISSUE} ]]; then - PATCH_FILE="${PATCH_OR_ISSUE}" + patchfile="${PATCH_OR_ISSUE}" else - if [[ ${PATCH_OR_ISSUE} =~ ^http ]]; then - echo "Patch is being downloaded at $(date) from" - PATCHURL="${PATCH_OR_ISSUE}" - else - ${WGET} -q -O "${PATCH_DIR}/jira" "http://issues.apache.org/jira/browse/${PATCH_OR_ISSUE}" - - if [[ $? != 0 ]];then - hadoop_error "ERROR: Unable to determine what ${PATCH_OR_ISSUE} may reference." - cleanup_and_exit 1 - fi - - if [[ $(${GREP} -c 'Patch Available' "${PATCH_DIR}/jira") == 0 ]] ; then - if [[ ${JENKINS} == true ]]; then - hadoop_error "ERROR: ${PATCH_OR_ISSUE} is not \"Patch Available\"." - cleanup_and_exit 1 - else - hadoop_error "WARNING: ${PATCH_OR_ISSUE} is not \"Patch Available\"." + # run through the bug systems. maybe they know? + for bugsys in ${BUGSYSTEMS}; do + if declare -f ${bugsys}_locate_patch >/dev/null 2>&1; then + "${bugsys}_locate_patch" "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" + if [[ $? == 0 ]]; then + guess_patch_file "${PATCH_DIR}/patch" + if [[ $? == 0 ]]; then + gotit=true + break; + fi fi fi + done - relativePatchURL=$(${GREP} -o '"/jira/secure/attachment/[0-9]*/[^"]*' "${PATCH_DIR}/jira" | ${GREP} -v -e 'htm[l]*$' | sort | tail -1 | ${GREP} -o '/jira/secure/attachment/[0-9]*/[^"]*') - PATCHURL="http://issues.apache.org${relativePatchURL}" - if [[ ! ${PATCHURL} =~ \.patch$ ]]; then - notSureIfPatch=true - fi - patchNum=$(echo "${PATCHURL}" | ${GREP} -o '[0-9]*/' | ${GREP} -o '[0-9]*') - echo "${ISSUE} patch is being downloaded at $(date) from" - fi - echo "${PATCHURL}" - add_jira_footer "Patch URL" "${PATCHURL}" - ${WGET} -q -O "${PATCH_DIR}/patch" "${PATCHURL}" - if [[ $? != 0 ]];then - hadoop_error "ERROR: ${PATCH_OR_ISSUE} could not be downloaded." - cleanup_and_exit 1 + # ok, none of the bug systems know. let's see how smart we are + if [[ ${gotit} == false ]]; then + generic_locate_patch "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" fi - PATCH_FILE="${PATCH_DIR}/patch" fi - if [[ ! -f "${PATCH_DIR}/patch" ]]; then - cp "${PATCH_FILE}" "${PATCH_DIR}/patch" + if [[ ! -f "${PATCH_DIR}/patch" + && -f "${patchfile}" ]]; then + cp "${patchfile}" "${PATCH_DIR}/patch" if [[ $? == 0 ]] ; then - echo "Patch file ${PATCH_FILE} copied to ${PATCH_DIR}" + echo "Patch file ${patchfile} copied to ${PATCH_DIR}" else - hadoop_error "ERROR: Could not copy ${PATCH_FILE} to ${PATCH_DIR}" + yetus_error "ERROR: Could not copy ${patchfile} to ${PATCH_DIR}" cleanup_and_exit 1 fi fi - if [[ ${notSureIfPatch} == "true" ]]; then - guess_patch_file "${PATCH_DIR}/patch" - if [[ $? != 0 ]]; then - hadoop_error "ERROR: ${PATCHURL} is not a patch file." - cleanup_and_exit 1 - else - hadoop_debug "The patch ${PATCHURL} was not named properly, but it looks like a patch file. proceeding, but issue/branch matching might go awry." - add_jira_table 0 patch "The patch file was not named according to ${PROJECT_NAME}'s naming conventions. Please see ${HOW_TO_CONTRIBUTE} for instructions." - fi + + guess_patch_file "${PATCH_DIR}/patch" + if [[ $? != 0 ]]; then + yetus_error "ERROR: Unsure how to process ${PATCH_OR_ISSUE}." + cleanup_and_exit 1 fi } @@ -1419,15 +1609,19 @@ function guess_patch_file local patch=$1 local fileOutput - hadoop_debug "Trying to guess is ${patch} is a patch file." + if [[ ! -f ${patch} ]]; then + return 1 + fi + + yetus_debug "Trying to guess is ${patch} is a patch file." fileOutput=$("${FILE}" "${patch}") if [[ $fileOutput =~ \ diff\ ]]; then - hadoop_debug "file magic says it's a diff." + yetus_debug "file magic says it's a diff." return 0 fi - fileOutput=$(head -n 1 "${patch}" | "${EGREP}" "^(From [a-z0-9]* Mon Sep 17 00:00:00 2001)|(diff .*)|(Index: .*)$") + fileOutput=$(head -n 1 "${patch}" | "${GREP}" -E "^(From [a-z0-9]* Mon Sep 17 00:00:00 2001)|(diff .*)|(Index: .*)$") if [[ $? == 0 ]]; then - hadoop_debug "first line looks like a patch file." + yetus_debug "first line looks like a patch file." return 0 fi return 1 @@ -1445,10 +1639,10 @@ function verify_patch_file # Before building, check to make sure that the patch is valid export PATCH - "${BINDIR}/smart-apply-patch.sh" "${PATCH_DIR}/patch" dryrun + "${BINDIR}/smart-apply-patch.sh" --dry-run "${PATCH_DIR}/patch" if [[ $? != 0 ]] ; then echo "PATCH APPLICATION FAILED" - add_jira_table -1 patch "The patch command could not apply the patch during dryrun." + add_vote_table -1 patch "The patch command could not apply the patch during dryrun." return 1 else return 0 @@ -1470,17 +1664,90 @@ function apply_patch_file if [[ $? != 0 ]] ; then echo "PATCH APPLICATION FAILED" ((RESULT = RESULT + 1)) - add_jira_table -1 patch "The patch command could not apply the patch." - output_to_console 1 - output_to_jira 1 + add_vote_table -1 patch "The patch command could not apply the patch." + bugsystem_finalreport 1 cleanup_and_exit 1 fi return 0 } +## @description copy the test-patch binary bits to a new working dir, +## @description setting USER_PLUGIN_DIR and PERSONALITY to the new +## @description locations. +## @description this is used for test-patch in docker and reexec mode +## @audience private +## @stability evolving +## @replaceable no +function copytpbits +{ + local dockerdir + local dockfile + local person + # we need to copy/consolidate all the bits that might have changed + # that are considered part of test-patch. This *might* break + # things that do off-path includes, but there isn't much we can + # do about that, I don't think. + + # if we've already copied, then don't bother doing it again + if [[ ${STARTDIR} == ${PATCH_DIR}/precommit ]]; then + yetus_debug "Skipping copytpbits; already copied once" + return + fi + + pushd "${STARTINGDIR}" >/dev/null + mkdir -p "${PATCH_DIR}/precommit/user-plugins" + mkdir -p "${PATCH_DIR}/precommit/personality" + mkdir -p "${PATCH_DIR}/precommit/test-patch-docker" + + # copy our entire universe, preserving links, etc. + (cd "${BINDIR}"; tar cpf - . ) | (cd "${PATCH_DIR}/precommit"; tar xpf - ) + + if [[ -n "${USER_PLUGIN_DIR}" + && -d "${USER_PLUGIN_DIR}" ]]; then + cp -pr "${USER_PLUGIN_DIR}/*" \ + "${PATCH_DIR}/precommit/user-plugins" + fi + # Set to be relative to ${PATCH_DIR}/precommit + USER_PLUGIN_DIR="${PATCH_DIR}/precommit/user-plugins" + + if [[ -n ${PERSONALITY} + && -f ${PERSONALITY} ]]; then + cp -pr "${PERSONALITY}" "${PATCH_DIR}/precommit/personality" + person=$(basename "${PERSONALITY}") + + # Set to be relative to ${PATCH_DIR}/precommit + PERSONALITY="${PATCH_DIR}/precommit/personality/${person}" + fi + + if [[ -n ${DOCKERFILE} + && -f ${DOCKERFILE} ]]; then + dockerdir=$(dirname "${DOCKERFILE}") + dockfile=$(basename "${DOCKERFILE}") + pushd "${dockerdir}" >/dev/null + gitfilerev=$("${GIT}" log -n 1 --pretty=format:%h -- "${dockfile}" 2>/dev/null) + popd >/dev/null + if [[ -z ${gitfilerev} ]]; then + gitfilerev=$(date "+%F") + gitfilerev="date${gitfilerev}" + fi + ( + echo "### TEST_PATCH_PRIVATE: dockerfile=${DOCKERFILE}" + echo "### TEST_PATCH_PRIVATE: gitrev=${gitfilerev}" + cat "${DOCKERFILE}" + # make sure we put some space between, just in case last + # line isn't an empty line or whatever + printf "\n\n" + cat "${BINDIR}/test-patch-docker/Dockerfile-endstub" + + printf "\n\n" + ) > "${PATCH_DIR}/precommit/test-patch-docker/Dockerfile" + DOCKERFILE="${PATCH_DIR}/precommit/test-patch-docker/Dockerfile" + fi + + popd >/dev/null +} -## @description If this actually patches the files used for the QA process -## @description under dev-support and its subdirectories, then +## @description If this patches actually patches test-patch.sh, then ## @description run with the patched version for the test. ## @audience private ## @stability evolving @@ -1489,657 +1756,775 @@ function apply_patch_file function check_reexec { local commentfile=${PATCH_DIR}/tp.${RANDOM} + local tpdir + local copy=false + local testdir + local person if [[ ${REEXECED} == true ]]; then big_console_header "Re-exec mode detected. Continuing." return fi - if [[ ! ${CHANGED_FILES} =~ dev-support/test-patch - && ! ${CHANGED_FILES} =~ dev-support/smart-apply ]] ; then - return - fi + for testdir in "${BINDIR}" \ + "${PERSONALITY}" \ + "${USER_PLUGIN_DIR}" \ + "${DOCKERFILE}"; do + tpdir=$(relative_dir "${testdir}") + if [[ $? == 0 + && ${CHANGED_FILES} =~ ${tpdir} ]]; then + copy=true + fi + done - big_console_header "dev-support patch detected" + if [[ ${copy} == true ]]; then + big_console_header "precommit patch detected" - if [[ ${RESETREPO} == false ]]; then - ((RESULT = RESULT + 1)) - hadoop_debug "can't destructively change the working directory. run with '--resetrepo' please. :(" - add_jira_table -1 dev-support "Couldn't test dev-support changes because we aren't configured to destructively change the working directory." + if [[ ${RESETREPO} == false ]]; then + ((RESULT = RESULT + 1)) + yetus_debug "can't destructively change the working directory. run with '--resetrepo' please. :(" + add_vote_table -1 precommit "Couldn't test precommit changes because we aren't configured to destructively change the working directory." + else + + apply_patch_file + + if [[ ${JENKINS} == true ]]; then + rm "${commentfile}" 2>/dev/null + echo "(!) A patch to the testing environment has been detected. " > "${commentfile}" + echo "Re-executing against the patched versions to perform further tests. " >> "${commentfile}" + echo "The console is at ${BUILD_URL}console in case of problems." >> "${commentfile}" + write_comment "${commentfile}" + rm "${commentfile}" + fi + fi + fi + + if [[ ${DOCKERSUPPORT} == false + && ${copy} == false ]]; then return fi - printf "\n\nRe-executing against patched versions to test.\n\n" + if [[ ${DOCKERSUPPORT} == true + && ${copy} == false ]]; then + big_console_header "Re-execing under Docker" - apply_patch_file + fi - if [[ ${JENKINS} == true ]]; then + # copy our universe + copytpbits - rm "${commentfile}" 2>/dev/null + if [[ ${DOCKERSUPPORT} == true ]]; then + # if we are doing docker, then we re-exec, but underneath the + # container - echo "(!) A patch to the files used for the QA process has been detected. " > "${commentfile}" - echo "Re-executing against the patched versions to perform further tests. " >> "${commentfile}" - echo "The console is at ${BUILD_URL}console in case of problems." >> "${commentfile}" + client=$(docker version | grep 'Client version' | cut -f2 -d: | tr -d ' ') + server=$(docker version | grep 'Server version' | cut -f2 -d: | tr -d ' ') - write_to_jira "${commentfile}" - rm "${commentfile}" - fi + dockerversion="Client=${client} Server=${server}" - cd "${CWD}" - mkdir -p "${PATCH_DIR}/dev-support-test" - (cd "${BINDIR}"; tar cpf - . ) \ - | (cd "${PATCH_DIR}/dev-support-test"; tar xpf - ) + TESTPATCHMODE="${USER_PARAMS[*]}" + if [[ -n "${BUILD_URL}" ]]; then + TESTPATCHMODE="--build-url=${BUILD_URL} ${TESTPATCHMODE}" + fi + TESTPATCHMODE="--tpglobaltimer=${GLOBALTIMER} ${TESTPATCHMODE}" + TESTPATCHMODE="--tpreexectimer=${TIMER} ${TESTPATCHMODE}" + TESTPATCHMODE="--personality=\'${PERSONALITY}\' ${TESTPATCHMODE}" + TESTPATCHMODE="--plugins=\'${USER_PLUGIN_DIR}\' ${TESTPATCHMODE}" + TESTPATCHMODE=" ${TESTPATCHMODE}" + export TESTPATCHMODE + + patchdir=$(relative_dir "${PATCH_DIR}") + + cd "${BASEDIR}" + #shellcheck disable=SC2093 + exec bash "${PATCH_DIR}/precommit/test-patch-docker/test-patch-docker.sh" \ + --dockerversion="${dockerversion}" \ + --java-home="${JAVA_HOME}" \ + --patch-dir="${patchdir}" \ + --project="${PROJECT_NAME}" - big_console_header "exec'ing test-patch.sh now..." + else - exec "${PATCH_DIR}/dev-support-test/test-patch.sh" \ - --reexec \ - --branch="${PATCH_BRANCH}" \ - --patch-dir="${PATCH_DIR}" \ - "${USER_PARAMS[@]}" + # if we aren't doing docker, then just call ourselves + # but from the new path with the new flags + #shellcheck disable=SC2093 + cd "${PATCH_DIR}/precommit/" + exec "${PATCH_DIR}/precommit/test-patch.sh" \ + "${USER_PARAMS[@]}" \ + --reexec \ + --basedir="${BASEDIR}" \ + --branch="${PATCH_BRANCH}" \ + --patch-dir="${PATCH_DIR}" \ + --tpglobaltimer="${GLOBALTIMER}" \ + --tpreexectimer="${TIMER}" \ + --personality="${PERSONALITY}" \ + --plugins="${USER_PLUGIN_DIR}" + fi } -## @description Check the current directory for @author tags -## @audience private +## @description Reset the test results +## @audience public ## @stability evolving ## @replaceable no -## @return 0 on success -## @return 1 on failure -function check_author +function modules_reset { - local authorTags - - big_console_header "Checking there are no @author tags in the patch." - - start_clock + MODULE_STATUS=() + MODULE_STATUS_TIMER=() + MODULE_STATUS_MSG=() + MODULE_STATUS_LOG=() +} - if [[ ${CHANGED_FILES} =~ dev-support/test-patch ]]; then - add_jira_table 0 @author "Skipping @author checks as test-patch has been patched." - return 0 +## @description Utility to print standard module errors +## @audience public +## @stability evolving +## @replaceable no +## @param repostatus +## @param testtype +## @param mvncmdline +function modules_messages +{ + local repostatus=$1 + local testtype=$2 + local summarymode=$3 + shift 2 + local modindex=0 + local repo + local goodtime=0 + local failure=false + local oldtimer + local statusjdk + local multijdkmode=false + + if [[ ${repostatus} == branch ]]; then + repo=${PATCH_BRANCH} + else + repo="the patch" fi - authorTags=$("${GREP}" -c -i '^[^-].*@author' "${PATCH_DIR}/patch") - echo "There appear to be ${authorTags} @author tags in the patch." - if [[ ${authorTags} != 0 ]] ; then - add_jira_table -1 @author \ - "The patch appears to contain ${authorTags} @author tags which the Hadoop" \ - " community has agreed to not allow in code contributions." - return 1 + verify_multijdk_test "${testtype}" + if [[ $? == 1 ]]; then + multijdkmode=true fi - add_jira_table +1 @author "The patch does not contain any @author tags." - return 0 -} -## @description Check the patch file for changed/new tests -## @audience private -## @stability evolving -## @replaceable no -## @return 0 on success -## @return 1 on failure -function check_modified_unittests -{ - local testReferences=0 - local i + oldtimer=${TIMER} - verify_needed_test unit + if [[ ${summarymode} == true + && ${ALLOWSUMMARIES} == true ]]; then - if [[ $? == 0 ]]; then - return 0 - fi + until [[ ${modindex} -eq ${#MODULE[@]} ]]; do - big_console_header "Checking there are new or changed tests in the patch." + if [[ ${multijdkmode} == true ]]; then + statusjdk=${MODULE_STATUS_JDK[${modindex}]} + fi - start_clock + if [[ "${MODULE_STATUS[${modindex}]}" == '+1' ]]; then + ((goodtime=goodtime + ${MODULE_STATUS_TIMER[${modindex}]})) + else + failure=true + start_clock + echo "" + echo "${MODULE_STATUS_MSG[${modindex}]}" + echo "" + offset_clock "${MODULE_STATUS_TIMER[${modindex}]}" + add_vote_table "${MODULE_STATUS[${modindex}]}" "${testtype}" "${MODULE_STATUS_MSG[${modindex}]}" + if [[ ${MODULE_STATUS[${modindex}]} == -1 + && -n "${MODULE_STATUS_LOG[${modindex}]}" ]]; then + add_footer_table "${testtype}" "@@BASE@@/${MODULE_STATUS_LOG[${modindex}]}" + fi + fi + ((modindex=modindex+1)) + done - for i in ${CHANGED_FILES}; do - if [[ ${i} =~ /test/ ]]; then - ((testReferences=testReferences + 1)) + if [[ ${failure} == false ]]; then + start_clock + offset_clock "${goodtime}" + add_vote_table +1 "${testtype}" "${repo} passed${statusjdk}" fi - done - - echo "There appear to be ${testReferences} test file(s) referenced in the patch." - if [[ ${testReferences} == 0 ]] ; then - add_jira_table -1 "tests included" \ - "The patch doesn't appear to include any new or modified tests. " \ - "Please justify why no new tests are needed for this patch." \ - "Also please list what manual steps were performed to verify this patch." - return 1 + else + until [[ ${modindex} -eq ${#MODULE[@]} ]]; do + start_clock + echo "" + echo "${MODULE_STATUS_MSG[${modindex}]}" + echo "" + offset_clock "${MODULE_STATUS_TIMER[${modindex}]}" + add_vote_table "${MODULE_STATUS[${modindex}]}" "${testtype}" "${MODULE_STATUS_MSG[${modindex}]}" + if [[ ${MODULE_STATUS[${modindex}]} == -1 + && -n "${MODULE_STATUS_LOG[${modindex}]}" ]]; then + add_footer_table "${testtype}" "@@BASE@@/${MODULE_STATUS_LOG[${modindex}]}" + fi + ((modindex=modindex+1)) + done fi - add_jira_table +1 "tests included" \ - "The patch appears to include ${testReferences} new or modified test files." - return 0 + TIMER=${oldtimer} } -## @description Helper for check_javadoc -## @audience private +## @description Add a test result +## @audience public ## @stability evolving ## @replaceable no -## @return 0 on success -## @return 1 on failure -function count_javadoc_warns +## @param module +## @param runtime +function module_status { - local warningfile=$1 + local index=$1 + local value=$2 + local log=$3 + shift 3 - #shellcheck disable=SC2016,SC2046 - return $(${EGREP} "^[0-9]+ warnings$" "${warningfile}" | ${AWK} '{sum+=$1} END {print sum}') + local jdk + + jdk=$(report_jvm_version "${JAVA_HOME}") + + if [[ -n ${index} + && ${index} =~ ^[0-9]+$ ]]; then + MODULE_STATUS[${index}]="${value}" + MODULE_STATUS_LOG[${index}]="${log}" + MODULE_STATUS_JDK[${index}]=" with JDK v${jdk}" + MODULE_STATUS_MSG[${index}]="${*}" + else + yetus_error "ASSERT: module_status given bad index: ${index}" + local frame=0 + while caller $frame; do + ((frame++)); + done + echo "$*" + exit 1 + fi } -## @description Count and compare the number of JavaDoc warnings pre- and post- patch -## @audience private +## @description run the tests for the queued modules +## @audience public ## @stability evolving ## @replaceable no -## @return 0 on success -## @return 1 on failure -function check_javadoc +## @param repostatus +## @param testtype +## @param mvncmdline +function modules_workers { - local numBranchJavadocWarnings - local numPatchJavadocWarnings + local repostatus=$1 + local testtype=$2 + shift 2 + local modindex=0 + local fn + local savestart=${TIMER} + local savestop + local repo + local modulesuffix + local jdk="" + local jdkindex=0 + local statusjdk + + if [[ ${repostatus} == branch ]]; then + repo=${PATCH_BRANCH} + else + repo="the patch" + fi - verify_needed_test javadoc + modules_reset - if [[ $? == 0 ]]; then - echo "This patch does not appear to need javadoc checks." - return 0 + verify_multijdk_test "${testtype}" + if [[ $? == 1 ]]; then + jdk=$(report_jvm_version "${JAVA_HOME}") + statusjdk=" with JDK v${jdk}" + jdk="-jdk${jdk}" + jdk=${jdk// /} + yetus_debug "Starting MultiJDK mode${statusjdk} on ${testtype}" fi - big_console_header "Determining number of patched javadoc warnings" + until [[ ${modindex} -eq ${#MODULE[@]} ]]; do + start_clock - start_clock + fn=$(module_file_fragment "${MODULE[${modindex}]}") + fn="${fn}${jdk}" + modulesuffix=$(basename "${MODULE[${modindex}]}") + pushd "${BASEDIR}/${MODULE[${modindex}]}" >/dev/null - if [[ -d hadoop-project ]]; then - (cd hadoop-project; "${MVN}" "${MAVEN_ARGS[@]}" install > /dev/null 2>&1) - fi - if [[ -d hadoop-common-project/hadoop-annotations ]]; then - (cd hadoop-common-project/hadoop-annotations; "${MVN}" "${MAVEN_ARGS[@]}" install > /dev/null 2>&1) - fi - echo_and_redirect "${PATCH_DIR}/patchJavadocWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean test javadoc:javadoc -DskipTests -Pdocs -D${PROJECT_NAME}PatchProcess - count_javadoc_warns "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarnings.txt" - numBranchJavadocWarnings=$? - count_javadoc_warns "${PATCH_DIR}/patchJavadocWarnings.txt" - numPatchJavadocWarnings=$? - - echo "There appear to be ${numBranchJavadocWarnings} javadoc warnings before the patch and ${numPatchJavadocWarnings} javadoc warnings after applying the patch." - if [[ ${numBranchJavadocWarnings} != "" && ${numPatchJavadocWarnings} != "" ]] ; then - if [[ ${numPatchJavadocWarnings} -gt ${numBranchJavadocWarnings} ]] ; then - - ${GREP} -i warning "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarnings.txt" > "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarningsFiltered.txt" - ${GREP} -i warning "${PATCH_DIR}/patchJavadocWarnings.txt" > "${PATCH_DIR}/patchJavadocWarningsFiltered.txt" - ${DIFF} -u "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarningsFiltered.txt" \ - "${PATCH_DIR}/patchJavadocWarningsFiltered.txt" \ - > "${PATCH_DIR}/diffJavadocWarnings.txt" - rm -f "${PATCH_DIR}/${PATCH_BRANCH}JavadocWarningsFiltered.txt" "${PATCH_DIR}/patchJavadocWarningsFiltered.txt" - - add_jira_table -1 javadoc "The applied patch generated "\ - "$((numPatchJavadocWarnings-numBranchJavadocWarnings))" \ - " additional warning messages." - add_jira_footer javadoc "@@BASE@@/diffJavadocWarnings.txt" - return 1 + if [[ ${modulesuffix} == . ]]; then + modulesuffix="root" fi - fi - add_jira_table +1 javadoc "There were no new javadoc warning messages." - return 0 -} -## @description Make sure site still compiles -## @audience private -## @stability evolving -## @replaceable no -## @return 0 on success -## @return 1 on failure -function check_site -{ - local -r mypwd=$(pwd) - - verify_needed_test site + if [[ $? != 0 ]]; then + echo "${BASEDIR}/${MODULE[${modindex}]} no longer exists. Skipping." + ((modindex=modindex+1)) + continue + fi - if [[ $? == 0 ]]; then - echo "This patch does not appear to need site checks." - return 0 - fi + echo_and_redirect "${PATCH_DIR}/${repostatus}-${testtype}-${fn}.txt" \ + $("${BUILDTOOL}_executor") \ + ${MODULEEXTRAPARAM[${modindex}]//@@@MODULEFN@@@/${fn}} \ + "${@//@@@MODULEFN@@@/${fn}}" - big_console_header "Determining if patched site still builds" + if [[ $? == 0 ]] ; then + module_status \ + ${modindex} \ + +1 \ + "${repostatus}-${testtype}-${fn}.txt" \ + "${modulesuffix} in ${repo} passed${statusjdk}." + else + module_status \ + ${modindex} \ + -1 \ + "${repostatus}-${testtype}-${fn}.txt" \ + "${modulesuffix} in ${repo} failed${statusjdk}." + ((result = result + 1)) + fi + savestop=$(stop_clock) + MODULE_STATUS_TIMER[${modindex}]=${savestop} + # shellcheck disable=SC2086 + echo "Elapsed: $(clock_display ${savestop})" + popd >/dev/null + ((modindex=modindex+1)) + done - start_clock + TIMER=${savestart} - echo "site creation for ${mypwd}" - echo_and_redirect "${PATCH_DIR}/patchSiteWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean site site:stage -DskipTests -Dmaven.javadoc.skip=true -D${PROJECT_NAME}PatchProcess - if [[ $? != 0 ]] ; then - echo "Site compilation is broken" - add_jira_table -1 site "Site compilation is broken." - add_jira_footer site "@@BASE@@/patchSiteWarnings.txt" + if [[ ${result} -gt 0 ]]; then return 1 fi - add_jira_table +1 site "Site still builds." return 0 } -## @description Helper for check_javac -## @audience private +## @description Reset the queue for tests +## @audience public ## @stability evolving ## @replaceable no -## @return 0 on success -## @return 1 on failure -function count_javac_warns +function clear_personality_queue { - local warningfile=$1 - #shellcheck disable=SC2016,SC2046 - return $(${AWK} 'BEGIN {total = 0} {total += 1} END {print total}' "${warningfile}") + yetus_debug "Personality: clear queue" + MODCOUNT=0 + MODULE=() } -## @description Count and compare the number of javac warnings pre- and post- patch -## @audience private +## @description Build the queue for tests +## @audience public ## @stability evolving ## @replaceable no +## @param module +## @param profiles/flags/etc +function personality_enqueue_module +{ + yetus_debug "Personality: enqueue $*" + local module=$1 + shift + + MODULE[${MODCOUNT}]=${module} + MODULEEXTRAPARAM[${MODCOUNT}]=${*} + ((MODCOUNT=MODCOUNT+1)) +} + +## @description Confirm compilation pre-patch +## @audience private +## @stability stable +## @replaceable no ## @return 0 on success ## @return 1 on failure -function check_javac +function precheck_javac { - local branchJavacWarnings - local patchJavacWarnings + local result=0 + local -r savejavahome=${JAVA_HOME} + local multijdkmode=false + local jdkindex=0 - verify_needed_test javac + big_console_header "Pre-patch ${PATCH_BRANCH} javac compilation" + verify_needed_test javac if [[ $? == 0 ]]; then - echo "This patch does not appear to need javac checks." - return 0 + echo "Patch does not appear to need javac tests." + return 0 fi - big_console_header "Determining number of patched javac warnings." - - start_clock - - echo_and_redirect "${PATCH_DIR}/patchJavacWarnings.txt" "${MVN}" "${MAVEN_ARGS[@]}" clean test -DskipTests -D${PROJECT_NAME}PatchProcess ${NATIVE_PROFILE} -Ptest-patch - if [[ $? != 0 ]] ; then - add_jira_table -1 javac "The patch appears to cause the build to fail." - return 2 + verify_multijdk_test javac + if [[ $? == 1 ]]; then + multijdkmode=true fi - ### Compare ${PATCH_BRANCH} and patch javac warning numbers - if [[ -f ${PATCH_DIR}/patchJavacWarnings.txt ]] ; then - ${GREP} '\[WARNING\]' "${PATCH_DIR}/${PATCH_BRANCH}JavacWarnings.txt" > "${PATCH_DIR}/filtered${PATCH_BRANCH}JavacWarnings.txt" - ${GREP} '\[WARNING\]' "${PATCH_DIR}/patchJavacWarnings.txt" > "${PATCH_DIR}/filteredPatchJavacWarnings.txt" - - count_javac_warns "${PATCH_DIR}/filtered${PATCH_BRANCH}JavacWarnings.txt" - branchJavacWarnings=$? - count_javac_warns "${PATCH_DIR}/filteredPatchJavacWarnings.txt" - patchJavacWarnings=$? - echo "There appear to be ${branchJavacWarnings} javac compiler warnings before the patch and ${patchJavacWarnings} javac compiler warnings after applying the patch." - if [[ ${patchJavacWarnings} != "" && ${branchJavacWarnings} != "" ]] ; then - if [[ ${patchJavacWarnings} -gt ${branchJavacWarnings} ]] ; then + for jdkindex in ${JDK_DIR_LIST}; do + if [[ ${multijdkmode} == true ]]; then + JAVA_HOME=${jdkindex} + fi - ${DIFF} "${PATCH_DIR}/filtered${PATCH_BRANCH}JavacWarnings.txt" \ - "${PATCH_DIR}/filteredPatchJavacWarnings.txt" \ - > "${PATCH_DIR}/diffJavacWarnings.txt" + personality_modules branch javac + "${BUILDTOOL}_modules_worker" branch javac - add_jira_table -1 javac "The applied patch generated "\ - "$((patchJavacWarnings-branchJavacWarnings))" \ - " additional warning messages." + ((result=result + $?)) + modules_messages branch javac true - add_jira_footer javac "@@BASE@@/diffJavacWarnings.txt" + done + JAVA_HOME=${savejavahome} - return 1 - fi - fi + if [[ ${result} -gt 0 ]]; then + return 1 fi - - add_jira_table +1 javac "There were no new javac warning messages." return 0 } -## @description Verify all files have an Apache License +## @description Confirm Javadoc pre-patch ## @audience private -## @stability evolving +## @stability stable ## @replaceable no ## @return 0 on success ## @return 1 on failure -function check_apachelicense +function precheck_javadoc { - big_console_header "Determining number of patched release audit warnings." + local result=0 + local -r savejavahome=${JAVA_HOME} + local multijdkmode=false + local jdkindex=0 - start_clock + big_console_header "Pre-patch ${PATCH_BRANCH} Javadoc verification" - echo_and_redirect "${PATCH_DIR}/patchReleaseAuditOutput.txt" "${MVN}" "${MAVEN_ARGS[@]}" apache-rat:check -D${PROJECT_NAME}PatchProcess - #shellcheck disable=SC2038 - find "${BASEDIR}" -name rat.txt | xargs cat > "${PATCH_DIR}/patchReleaseAuditWarnings.txt" + verify_needed_test javadoc + if [[ $? == 0 ]]; then + echo "Patch does not appear to need javadoc tests." + return 0 + fi - ### Compare ${PATCH_BRANCH} and patch release audit warning numbers - if [[ -f ${PATCH_DIR}/patchReleaseAuditWarnings.txt ]] ; then - patchReleaseAuditWarnings=$("${GREP}" -c '\!?????' "${PATCH_DIR}/patchReleaseAuditWarnings.txt") - echo "" - echo "" - echo "There appear to be ${patchReleaseAuditWarnings} release audit warnings after applying the patch." - if [[ ${patchReleaseAuditWarnings} != "" ]] ; then - if [[ ${patchReleaseAuditWarnings} -gt 0 ]] ; then - add_jira_table -1 "release audit" "The applied patch generated ${patchReleaseAuditWarnings} release audit warnings." + verify_multijdk_test javadoc + if [[ $? == 1 ]]; then + multijdkmode=true + fi - ${GREP} '\!?????' "${PATCH_DIR}/patchReleaseAuditWarnings.txt" \ - > "${PATCH_DIR}/patchReleaseAuditProblems.txt" + for jdkindex in ${JDK_DIR_LIST}; do + if [[ ${multijdkmode} == true ]]; then + JAVA_HOME=${jdkindex} + fi - echo "Lines that start with ????? in the release audit "\ - "report indicate files that do not have an Apache license header." \ - >> "${PATCH_DIR}/patchReleaseAuditProblems.txt" + personality_modules branch javadoc + ${BUILDTOOL}_modules_worker branch javadoc - add_jira_footer "Release Audit" "@@BASE@@/patchReleaseAuditProblems.txt" + ((result=result + $?)) + modules_messages branch javadoc true - return 1 - fi - fi + done + JAVA_HOME=${savejavahome} + + if [[ ${result} -gt 0 ]]; then + return 1 fi - add_jira_table 1 "release audit" "The applied patch does not increase the total number of release audit warnings." return 0 } -## @description Verify mvn install works +## @description Confirm the source environment pre-patch ## @audience private -## @stability evolving +## @stability stable ## @replaceable no ## @return 0 on success ## @return 1 on failure -function check_mvn_install +function precheck_without_patch { - local retval + local result=0 - verify_needed_test javadoc - retval=$? + if declare -f ${BUILDTOOL}_precheck_install >/dev/null; then + "${BUILDTOOL}_precheck_install" + if [[ $? -gt 0 ]]; then + ((result = result +1 )) + fi + fi - verify_needed_test javac - ((retval = retval + $? )) + precheck_javac - if [[ ${retval} == 0 ]]; then - echo "This patch does not appear to need mvn install checks." - return 0 + if [[ $? -gt 0 ]]; then + ((result = result +1 )) fi - big_console_header "Installing all of the jars" + precheck_javadoc - start_clock - echo_and_redirect "${PATCH_DIR}/jarinstall.txt" "${MVN}" "${MAVEN_ARGS[@]}" install -Dmaven.javadoc.skip=true -DskipTests -D${PROJECT_NAME}PatchProcess - retval=$? - if [[ ${retval} != 0 ]]; then - add_jira_table -1 install "The patch causes mvn install to fail." - else - add_jira_table +1 install "mvn install still works." + if [[ $? -gt 0 ]]; then + ((result = result +1 )) fi - return ${retval} -} -## @description are the needed bits for findbugs present? -## @audience private -## @stability evolving -## @replaceable no -## @return 0 findbugs will work for our use -## @return 1 findbugs is missing some component -function findbugs_is_installed -{ - if [[ ! -e "${FINDBUGS_HOME}/bin/findbugs" ]]; then - printf "\n\n%s is not executable.\n\n" "${FINDBUGS_HOME}/bin/findbugs" - add_jira_table -1 findbugs "Findbugs is not installed." + if [[ ${result} -gt 0 ]]; then return 1 fi + return 0 } -## @description Run the maven findbugs plugin and record found issues in a bug database +## @description Check the current directory for @author tags ## @audience private ## @stability evolving ## @replaceable no ## @return 0 on success ## @return 1 on failure -function findbugs_mvnrunner +function check_author { - local name=$1 - local logfile=$2 - local warnings_file=$3 + local authorTags + local -r appname=$(basename "${BASH_SOURCE-$0}") - echo_and_redirect "${logfile}" "${MVN}" "${MAVEN_ARGS[@]}" clean test findbugs:findbugs -DskipTests \ - "-D${PROJECT_NAME}PatchProcess" < /dev/null - if [[ $? != 0 ]]; then - return 1 - fi - cp target/findbugsXml.xml "${warnings_file}.xml" + big_console_header "Checking there are no @author tags in the patch." - "${FINDBUGS_HOME}/bin/setBugDatabaseInfo" -name "${name}" \ - "${warnings_file}.xml" "${warnings_file}.xml" - if [[ $? != 0 ]]; then - return 1 + start_clock + + if [[ ${CHANGED_FILES} =~ ${appname} ]]; then + echo "Skipping @author checks as ${appname} has been patched." + add_vote_table 0 @author "Skipping @author checks as ${appname} has been patched." + return 0 fi - "${FINDBUGS_HOME}/bin/convertXmlToText" -html "${warnings_file}.xml" \ - "${warnings_file}.html" - if [[ $? != 0 ]]; then + authorTags=$("${GREP}" -c -i '^[^-].*@author' "${PATCH_DIR}/patch") + echo "There appear to be ${authorTags} @author tags in the patch." + if [[ ${authorTags} != 0 ]] ; then + add_vote_table -1 @author \ + "The patch appears to contain ${authorTags} @author tags which the" \ + " community has agreed to not allow in code contributions." return 1 fi - + add_vote_table +1 @author "The patch does not contain any @author tags." return 0 } -## @description Track pre-existing findbugs warnings +## @description Check the patch file for changed/new tests ## @audience private ## @stability evolving ## @replaceable no ## @return 0 on success ## @return 1 on failure -function precheck_findbugs +function check_modified_unittests { - local -r mypwd=$(pwd) - local module_suffix - local modules=${CHANGED_MODULES} - local module - local findbugs_version - local rc=0 - local module_findbugs_warnings - local findbugs_warnings=0 + local testReferences=0 + local i + + big_console_header "Checking there are new or changed tests in the patch." - verify_needed_test findbugs + verify_needed_test unit if [[ $? == 0 ]]; then - echo "Patch does not appear to need findbugs tests." + echo "Patch does not appear to need new or modified tests." return 0 fi - echo "findbugs baseline for ${mypwd}" - - findbugs_is_installed - if [[ $? != 0 ]]; then - return 1 - fi - - for module in ${modules} - do - pushd "${module}" >/dev/null - echo " Running findbugs in ${module}" - module_suffix=$(basename "${module}") - findbugs_mvnrunner "${PATCH_BRANCH}" \ - "${PATCH_DIR}/${PATCH_BRANCH}FindBugsOutput${module_suffix}.txt" \ - "${PATCH_DIR}/${PATCH_BRANCH}FindbugsWarnings${module_suffix}" - (( rc = rc + $? )) - - if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" ]]; then - #shellcheck disable=SC2016 - module_findbugs_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -first \ - "${PATCH_BRANCH}" \ - "${PATCH_DIR}/${PATCH_BRANCH}FindbugsWarnings${module_suffix}".xml \ - "${PATCH_DIR}/${PATCH_BRANCH}FindbugsWarnings${module_suffix}".xml \ - | ${AWK} '{print $1}') - if [[ $? != 0 ]]; then - popd >/dev/null - return 1 - fi - - findbugs_warnings=$((findbugs_warnings+module_findbugs_warnings)) + start_clock - if [[ ${module_findbugs_warnings} -gt 0 ]] ; then - add_jira_footer "Pre-patch Findbugs warnings" "@@BASE@@/${PATCH_BRANCH}FindbugsWarnings${module_suffix}.html" - fi + for i in ${CHANGED_FILES}; do + if [[ ${i} =~ (^|/)test/ ]]; then + ((testReferences=testReferences + 1)) fi - popd >/dev/null done - #shellcheck disable=SC2016 - findbugs_version=$(${AWK} 'match($0, /findbugs-maven-plugin:[^:]*:findbugs/) { print substr($0, RSTART + 22, RLENGTH - 31); exit }' "${PATCH_DIR}/${PATCH_BRANCH}FindBugsOutput${module_suffix}.txt") - - if [[ ${rc} -ne 0 ]]; then - echo "Pre-patch ${PATCH_BRANCH} findbugs is broken?" - add_jira_table -1 pre-patch "Findbugs (version ${findbugs_version}) appears to be broken on ${PATCH_BRANCH}." - return 1 - fi - - if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" && \ - ${findbugs_warnings} -gt 0 ]] ; then - echo "Pre-patch ${PATCH_BRANCH} findbugs has ${findbugs_warnings} warnings." - add_jira_table -1 pre-patch "Pre-patch ${PATCH_BRANCH} has ${findbugs_warnings} extant Findbugs (version ${findbugs_version}) warnings." + echo "There appear to be ${testReferences} test file(s) referenced in the patch." + if [[ ${testReferences} == 0 ]] ; then + add_vote_table -1 "test4tests" \ + "The patch doesn't appear to include any new or modified tests. " \ + "Please justify why no new tests are needed for this patch." \ + "Also please list what manual steps were performed to verify this patch." return 1 fi - + add_vote_table +1 "test4tests" \ + "The patch appears to include ${testReferences} new or modified test files." return 0 } -## @description Verify patch does not trigger any findbugs warnings +## @description Count and compare the number of javac warnings pre- and post- patch ## @audience private ## @stability evolving ## @replaceable no ## @return 0 on success ## @return 1 on failure -function check_findbugs +function check_patch_javac { - local rc=0 - local module - local modules=${CHANGED_MODULES} - local module_suffix - local combined_xml - local newBugs - local new_findbugs_warnings - local new_findbugs_fixed_warnings - local findbugs_warnings=0 - local findbugs_fixed_warnings=0 - local line - local firstpart - local secondpart - local findbugs_version + local i + local result=0 + local fn + local -r savejavahome=${JAVA_HOME} + local multijdkmode=false + local jdk="" + local jdkindex=0 + local statusjdk + declare -i numbranch=0 + declare -i numpatch=0 + + big_console_header "Determining number of patched javac errors" - verify_needed_test findbugs + verify_needed_test javac if [[ $? == 0 ]]; then + echo "Patch does not appear to need javac tests." return 0 fi - big_console_header "Determining number of patched Findbugs warnings." - - start_clock - - findbugs_is_installed - if [[ $? != 0 ]]; then - return 1 + verify_multijdk_test javac + if [[ $? == 1 ]]; then + multijdkmode=true fi - for module in ${modules} - do - pushd "${module}" >/dev/null - echo " Running findbugs in ${module}" - module_suffix=$(basename "${module}") - - findbugs_mvnrunner patch \ - "${PATCH_DIR}/patchFindBugsOutput${module_suffix}.txt" \ - "${PATCH_DIR}/patchFindbugsWarnings${module_suffix}" - - if [[ $? != 0 ]] ; then - ((rc = rc +1)) - echo "Post-patch findbugs compilation is broken." - add_jira_table -1 findbugs "Post-patch findbugs ${module} compilation is broken." - continue + for jdkindex in ${JDK_DIR_LIST}; do + if [[ ${multijdkmode} == true ]]; then + JAVA_HOME=${jdkindex} + jdk=$(report_jvm_version "${JAVA_HOME}") + yetus_debug "Using ${JAVA_HOME} to run this set of tests" + statusjdk=" with JDK v${jdk}" + jdk="-jdk${jdk}" + jdk=${jdk// /} fi - combined_xml="$PATCH_DIR/combinedFindbugsWarnings${module_suffix}.xml" - newBugs="${PATCH_DIR}/newPatchFindbugsWarnings${module_suffix}" - "${FINDBUGS_HOME}/bin/computeBugHistory" -useAnalysisTimes -withMessages \ - -output "${combined_xml}" \ - "${PATCH_DIR}/${PATCH_BRANCH}FindbugsWarnings${module_suffix}.xml" \ - "${PATCH_DIR}/patchFindbugsWarnings${module_suffix}.xml" - if [[ $? != 0 ]]; then - popd >/dev/null - return 1 - fi + personality_modules patch javac + ${BUILDTOOL}_modules_worker patch javac - #shellcheck disable=SC2016 - new_findbugs_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -first patch \ - "${combined_xml}" "${newBugs}.xml" | ${AWK} '{print $1}') - if [[ $? != 0 ]]; then - popd >/dev/null - return 1 - fi - #shellcheck disable=SC2016 - new_findbugs_fixed_warnings=$("${FINDBUGS_HOME}/bin/filterBugs" -fixed patch \ - "${combined_xml}" "${newBugs}.xml" | ${AWK} '{print $1}') - if [[ $? != 0 ]]; then - popd >/dev/null - return 1 - fi + i=0 + until [[ ${i} -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi - echo "Found ${new_findbugs_warnings} new Findbugs warnings and ${new_findbugs_fixed_warnings} newly fixed warnings." - findbugs_warnings=$((findbugs_warnings+new_findbugs_warnings)) - findbugs_fixed_warnings=$((findbugs_fixed_warnings+new_findbugs_fixed_warnings)) + fn=$(module_file_fragment "${MODULE[${i}]}") + fn="${fn}${jdk}" + module_suffix=$(basename "${MODULE[${i}]}") + if [[ ${module_suffix} == \. ]]; then + module_suffix=root + fi - "${FINDBUGS_HOME}/bin/convertXmlToText" -html "${newBugs}.xml" \ - "${newBugs}.html" - if [[ $? != 0 ]]; then - popd >/dev/null - return 1 - fi + # if it was a new module, this won't exist. + if [[ -f "${PATCH_DIR}/branch-javac-${fn}.txt" ]]; then + ${GREP} -i warning "${PATCH_DIR}/branch-javac-${fn}.txt" \ + > "${PATCH_DIR}/branch-javac-${fn}-warning.txt" + else + touch "${PATCH_DIR}/branch-javac-${fn}.txt" \ + "${PATCH_DIR}/branch-javac-${fn}-warning.txt" + fi - if [[ ${new_findbugs_warnings} -gt 0 ]] ; then - populate_test_table FindBugs "module:${module_suffix}" - while read line; do - firstpart=$(echo "${line}" | cut -f2 -d:) - secondpart=$(echo "${line}" | cut -f9- -d' ') - add_jira_test_table "" "${firstpart}:${secondpart}" - done < <("${FINDBUGS_HOME}/bin/convertXmlToText" "${newBugs}.xml") + if [[ -f "${PATCH_DIR}/patch-javac-${fn}.txt" ]]; then + ${GREP} -i warning "${PATCH_DIR}/patch-javac-${fn}.txt" \ + > "${PATCH_DIR}/patch-javac-${fn}-warning.txt" + else + touch "${PATCH_DIR}/patch-javac-${fn}.txt" \ + "${PATCH_DIR}/patch-javac-${fn}-warning.txt" + fi - add_jira_footer "Findbugs warnings" "@@BASE@@/newPatchFindbugsWarnings${module_suffix}.html" - fi + numbranch=$(${BUILDTOOL}_count_javac_probs "${PATCH_DIR}/branch-javac-${fn}-warning.txt") + numpatch=$(${BUILDTOOL}_count_javac_probs "${PATCH_DIR}/patch-javac-${fn}-warning.txt") - popd >/dev/null - done + if [[ -n ${numbranch} + && -n ${numpatch} + && ${numpatch} -gt ${numbranch} ]]; then - #shellcheck disable=SC2016 - findbugs_version=$(${AWK} 'match($0, /findbugs-maven-plugin:[^:]*:findbugs/) { print substr($0, RSTART + 22, RLENGTH - 31); exit }' "${PATCH_DIR}/patchFindBugsOutput${module_suffix}.txt") + ${DIFF} -u "${PATCH_DIR}/branch-javac-${fn}-warning.txt" \ + "${PATCH_DIR}/patch-javac-${fn}-warning.txt" \ + > "${PATCH_DIR}/javac-${fn}-diff.txt" - if [[ ${findbugs_warnings} -gt 0 ]] ; then - add_jira_table -1 findbugs "The patch appears to introduce ${findbugs_warnings} new Findbugs (version ${findbugs_version}) warnings." - return 1 - fi + module_status ${i} -1 "javac-${fn}-diff.txt" \ + "Patched ${module_suffix} generated "\ + "$((numpatch-numbranch)) additional warning messages${statusjdk}." \ - if [[ ${findbugs_fixed_warnings} -gt 0 ]] ; then - add_jira_table +1 findbugs "The patch does not introduce any new Findbugs (version ${findbugs_version}) warnings, and fixes ${findbugs_fixed_warnings} pre-existing warnings." - else - add_jira_table +1 findbugs "The patch does not introduce any new Findbugs (version ${findbugs_version}) warnings." + ((result=result+1)) + fi + ((i=i+1)) + done + + modules_messages patch javac true + done + JAVA_HOME=${savejavahome} + + if [[ ${result} -gt 0 ]]; then + return 1 fi return 0 } -## @description Make sure Maven's eclipse generation works. +## @description Count and compare the number of JavaDoc warnings pre- and post- patch ## @audience private ## @stability evolving ## @replaceable no ## @return 0 on success ## @return 1 on failure -function check_mvn_eclipse +function check_patch_javadoc { - big_console_header "Running mvn eclipse:eclipse." + local i + local result=0 + local fn + local -r savejavahome=${JAVA_HOME} + local multijdkmode=false + local jdk="" + local jdkindex=0 + local statusjdk + declare -i numbranch=0 + declare -i numpatch=0 - verify_needed_test javac - if [[ $? == 0 ]]; then - echo "Patch does not touch any java files. Skipping mvn eclipse:eclipse" + big_console_header "Determining number of patched javadoc warnings" + + verify_needed_test javadoc + if [[ $? == 0 ]]; then + echo "Patch does not appear to need javadoc tests." return 0 fi - start_clock + verify_multijdk_test javadoc + if [[ $? == 1 ]]; then + multijdkmode=true + fi - echo_and_redirect "${PATCH_DIR}/patchEclipseOutput.txt" "${MVN}" "${MAVEN_ARGS[@]}" eclipse:eclipse -D${PROJECT_NAME}PatchProcess - if [[ $? != 0 ]] ; then - add_jira_table -1 eclipse:eclipse "The patch failed to build with eclipse:eclipse." + for jdkindex in ${JDK_DIR_LIST}; do + if [[ ${multijdkmode} == true ]]; then + JAVA_HOME=${jdkindex} + jdk=$(report_jvm_version "${JAVA_HOME}") + yetus_debug "Using ${JAVA_HOME} to run this set of tests" + statusjdk=" with JDK v${jdk}" + jdk="-jdk${jdk}" + jdk=${jdk// /} + fi + + personality_modules patch javadoc + ${BUILDTOOL}_modules_worker patch javadoc + + i=0 + until [[ ${i} -eq ${#MODULE[@]} ]]; do + if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then + ((result=result+1)) + ((i=i+1)) + continue + fi + + fn=$(module_file_fragment "${MODULE[${i}]}") + fn="${fn}${jdk}" + module_suffix=$(basename "${MODULE[${i}]}") + if [[ ${module_suffix} == \. ]]; then + module_suffix=root + fi + + if [[ -f "${PATCH_DIR}/branch-javadoc-${fn}.txt" ]]; then + ${GREP} -i warning "${PATCH_DIR}/branch-javadoc-${fn}.txt" \ + > "${PATCH_DIR}/branch-javadoc-${fn}-warning.txt" + else + touch "${PATCH_DIR}/branch-javadoc-${fn}.txt" \ + "${PATCH_DIR}/branch-javadoc-${fn}-warning.txt" + fi + + if [[ -f "${PATCH_DIR}/patch-javadoc-${fn}.txt" ]]; then + ${GREP} -i warning "${PATCH_DIR}/patch-javadoc-${fn}.txt" \ + > "${PATCH_DIR}/patch-javadoc-${fn}-warning.txt" + else + touch "${PATCH_DIR}/patch-javadoc-${fn}.txt" \ + "${PATCH_DIR}/patch-javadoc-${fn}-warning.txt" + fi + + numbranch=$(${BUILDTOOL}_count_javadoc_probs "${PATCH_DIR}/branch-javadoc-${fn}.txt") + numpatch=$(${BUILDTOOL}_count_javadoc_probs "${PATCH_DIR}/patch-javadoc-${fn}.txt") + + if [[ -n ${numbranch} + && -n ${numpatch} + && ${numpatch} -gt ${numbranch} ]] ; then + + ${DIFF} -u "${PATCH_DIR}/branch-javadoc-${fn}-warning.txt" \ + "${PATCH_DIR}/patch-javadoc-${fn}-warning.txt" \ + > "${PATCH_DIR}/javadoc-${fn}-diff.txt" + + module_status ${i} -1 "javadoc-${fn}-diff.txt" \ + "Patched ${module_suffix} generated "\ + "$((numpatch-numbranch)) additional warning messages${statusjdk}." + + ((result=result+1)) + fi + ((i=i+1)) + done + + modules_messages patch javadoc true + done + JAVA_HOME=${savejavahome} + + if [[ ${result} -gt 0 ]]; then return 1 fi - add_jira_table +1 eclipse:eclipse "The patch built with eclipse:eclipse." return 0 } @@ -2158,10 +2543,10 @@ function populate_test_table for i in "$@"; do if [[ -z "${first}" ]]; then - add_jira_test_table "${reason}" "${i}" + add_test_table "${reason}" "${i}" first="${reason}" else - add_jira_test_table " " "${i}" + add_test_table " " "${i}" fi done } @@ -2174,332 +2559,155 @@ function populate_test_table ## @return 1 on failure function check_unittests { + local i + local testsys + local test_logfile + local result=0 + local -r savejavahome=${JAVA_HOME} + local multijdkmode=false + local jdk="" + local jdkindex=0 + local statusjdk + local formatresult=0 + local needlog + local unitlogs + verify_needed_test unit if [[ $? == 0 ]]; then - echo "Existing unit tests do not test patched files. Skipping." return 0 fi big_console_header "Running unit tests" - start_clock + verify_multijdk_test unit + if [[ $? == 1 ]]; then + multijdkmode=true + fi - local failed_tests="" - local modules=${CHANGED_MODULES} - local building_common=0 - local hdfs_modules - local ordered_modules="" - local failed_test_builds="" - local test_timeouts="" - local test_logfile - local test_build_result - local module_test_timeouts="" - local result - local totalresult=0 - local module_prefix - - # - # If we are building hadoop-hdfs-project, we must build the native component - # of hadoop-common-project first. In order to accomplish this, we move the - # hadoop-hdfs subprojects to the end of the list so that common will come - # first. - # - # Of course, we may not be building hadoop-common at all-- in this case, we - # explicitly insert a mvn compile -Pnative of common, to ensure that the - # native libraries show up where we need them. - # - - for module in ${modules}; do - if [[ ${module} == hadoop-hdfs-project* ]]; then - hdfs_modules="${hdfs_modules} ${module}" - elif [[ ${module} == hadoop-common-project* ]]; then - ordered_modules="${ordered_modules} ${module}" - building_common=1 - else - ordered_modules="${ordered_modules} ${module}" + for jdkindex in ${JDK_DIR_LIST}; do + if [[ ${multijdkmode} == true ]]; then + JAVA_HOME=${jdkindex} + jdk=$(report_jvm_version "${JAVA_HOME}") + statusjdk="JDK v${jdk} " + jdk="-jdk${jdk}" + jdk=${jdk// /} fi - done - if [[ -n "${hdfs_modules}" ]]; then - ordered_modules="${ordered_modules} ${hdfs_modules}" - if [[ ${building_common} -eq 0 ]]; then - echo " Building hadoop-common with -Pnative in order to provide libhadoop.so to the hadoop-hdfs unit tests." - echo_and_redirect "${PATCH_DIR}/testrun_native.txt" "${MVN}" "${MAVEN_ARGS[@]}" compile ${NATIVE_PROFILE} "-D${PROJECT_NAME}PatchProcess" - if [[ $? != 0 ]]; then - add_jira_table -1 "native" "Failed to build the native portion " \ - "of hadoop-common prior to running the unit tests in ${ordered_modules}" - return 1 - else - add_jira_table +1 "native" "Pre-build of native portion" - fi - fi - fi + personality_modules patch unit + ${BUILDTOOL}_modules_worker patch unit - for module in ${ordered_modules}; do - result=0 - start_clock - pushd "${module}" >/dev/null - module_suffix=$(basename "${module}") - module_prefix=$(echo "${module}" | cut -f2 -d- ) - - test_logfile=${PATCH_DIR}/testrun_${module_suffix}.txt - echo " Running tests in ${module_suffix}" - echo_and_redirect "${test_logfile}" "${MVN}" "${MAVEN_ARGS[@]}" clean install -fae ${NATIVE_PROFILE} ${REQUIRE_TEST_LIB_HADOOP} -D${PROJECT_NAME}PatchProcess - test_build_result=$? - - add_jira_footer "${module_suffix} test log" "@@BASE@@/testrun_${module_suffix}.txt" - - # shellcheck disable=2016 - module_test_timeouts=$(${AWK} '/^Running / { if (last) { print last } last=$2 } /^Tests run: / { last="" }' "${test_logfile}") - if [[ -n "${module_test_timeouts}" ]] ; then - test_timeouts="${test_timeouts} ${module_test_timeouts}" - result=1 + ((result=result+$?)) + + modules_messages patch unit false + if [[ ${result} == 0 ]]; then + continue fi - #shellcheck disable=SC2026,SC2038,SC2016 - module_failed_tests=$(find . -name 'TEST*.xml'\ - | xargs "${GREP}" -l -E "/dev/null + + needlog=0 + for testsys in ${TESTFORMATS}; do + if declare -f ${testsys}_process_tests >/dev/null; then + yetus_debug "Calling ${testsys}_process_tests" + "${testsys}_process_tests" "${module}" "${test_logfile}" "${fn}" + formatresult=$? + ((results=results+formatresult)) + if [[ "${formatresult}" != 0 ]]; then + needlog=1 + fi + fi + done - if [[ -n "${module_failed_tests}" ]] ; then - failed_tests="${failed_tests} ${module_failed_tests}" - result=1 - fi - if [[ ${test_build_result} != 0 && -z "${module_failed_tests}" && -z "${module_test_timeouts}" ]] ; then - failed_test_builds="${failed_test_builds} ${module_suffix}" - result=1 - fi - popd >/dev/null + if [[ ${needlog} == 1 ]]; then + unitlogs="${unitlogs} @@BASE@@/patch-unit-${fn}.txt" + fi - if [[ $result == 1 ]]; then - add_jira_table -1 "${module_prefix} tests" "Tests failed in ${module_suffix}." - else - add_jira_table +1 "${module_prefix} tests" "Tests passed in ${module_suffix}." - fi + popd >/dev/null + + ((i=i+1)) + done - ((totalresult = totalresult + result)) done + JAVA_HOME=${savejavahome} - if [[ -n "${failed_tests}" ]] ; then - # shellcheck disable=SC2086 - populate_test_table "Failed unit tests" ${failed_tests} - fi - if [[ -n "${test_timeouts}" ]] ; then - # shellcheck disable=SC2086 - populate_test_table "Timed out tests" ${test_timeouts} - fi - if [[ -n "${failed_test_builds}" ]] ; then - # shellcheck disable=SC2086 - populate_test_table "Failed build" ${failed_test_builds} + if [[ -n "${unitlogs}" ]]; then + add_footer_table "unit test logs" "${unitlogs}" fi if [[ ${JENKINS} == true ]]; then - add_jira_footer "Test Results" "${BUILD_URL}testReport/" + add_footer_table "${statusjdk} Test Results" "${BUILD_URL}testReport/" fi - if [[ ${totalresult} -gt 0 ]]; then + for testsys in ${TESTFORMATS}; do + if declare -f ${testsys}_finalize_results >/dev/null; then + yetus_debug "Calling ${testsys}_finalize_results" + "${testsys}_finalize_results" "${statusjdk}" + fi + done + + if [[ ${result} -gt 0 ]]; then return 1 - else - return 0 fi + return 0 } -## @description Print out the finished details on the console -## @audience private +## @description Write comments onto bug systems that have code review support. +## @description File should be in the form of "file:line:comment" +## @audience public ## @stability evolving ## @replaceable no -## @param runresult -## @return 0 on success -## @return 1 on failure -function output_to_console -{ - local result=$1 - shift - local i - local ourstring - local vote - local subs - local ela - local comment - local commentfile1="${PATCH_DIR}/comment.1" - local commentfile2="${PATCH_DIR}/comment.2" - local normaltop - local line - local seccoladj=0 - local spcfx=${PATCH_DIR}/spcl.txt - - if [[ ${result} == 0 ]]; then - if [[ ${JENKINS} == false ]]; then - { - printf "IF9fX19fX19fX18gCjwgU3VjY2VzcyEgPgogLS0tLS0tLS0tLSAKIFwgICAg"; - printf "IC9cICBfX18gIC9cCiAgXCAgIC8vIFwvICAgXC8gXFwKICAgICAoKCAgICBP"; - printf "IE8gICAgKSkKICAgICAgXFwgLyAgICAgXCAvLwogICAgICAgXC8gIHwgfCAg"; - printf "XC8gCiAgICAgICAgfCAgfCB8ICB8ICAKICAgICAgICB8ICB8IHwgIHwgIAog"; - printf "ICAgICAgIHwgICBvICAgfCAgCiAgICAgICAgfCB8ICAgfCB8ICAKICAgICAg"; - printf "ICB8bXwgICB8bXwgIAo" - } > "${spcfx}" - fi - printf "\n\n+1 overall\n\n" - else - if [[ ${JENKINS} == false ]]; then - { - printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf"; - printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf"; - printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98"; - printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg"; - printf "ICAgICAgICAgICAgICAgICAK" - } > "${spcfx}" - fi - printf "\n\n-1 overall\n\n" - fi - - if [[ -f ${spcfx} ]]; then - if which base64 >/dev/null 2>&1; then - base64 --decode "${spcfx}" 2>/dev/null - elif which openssl >/dev/null 2>&1; then - openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null - fi - echo - echo - rm "${spcfx}" - fi - - seccoladj=$(findlargest 2 "${JIRA_COMMENT_TABLE[@]}") - if [[ ${seccoladj} -lt 10 ]]; then - seccoladj=10 +## @param filename +function bugsystem_linecomments +{ + declare title=$1 + declare fn=$2 + declare line + declare bugs + declare realline + declare text + declare idxline + declare uniline + + if [[ ! -f "${GITUNIDIFFLINES}" ]]; then + return fi - seccoladj=$((seccoladj + 2 )) - i=0 - until [[ $i -eq ${#JIRA_HEADER[@]} ]]; do - printf "%s\n" "${JIRA_HEADER[${i}]}" - ((i=i+1)) - done - - printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment" - echo "============================================================================" - i=0 - until [[ $i -eq ${#JIRA_COMMENT_TABLE[@]} ]]; do - ourstring=$(echo "${JIRA_COMMENT_TABLE[${i}]}" | tr -s ' ') - vote=$(echo "${ourstring}" | cut -f2 -d\|) - vote=$(colorstripper "${vote}") - subs=$(echo "${ourstring}" | cut -f3 -d\|) - ela=$(echo "${ourstring}" | cut -f4 -d\|) - comment=$(echo "${ourstring}" | cut -f5 -d\|) - - echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}" - normaltop=$(head -1 "${commentfile1}") - ${SED} -e '1d' "${commentfile1}" > "${commentfile2}" - - printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \ - "${subs}" "${ela}" "${normaltop}" - while read line; do - printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}" - done < "${commentfile2}" - - ((i=i+1)) - rm "${commentfile2}" "${commentfile1}" 2>/dev/null - done + while read -r line;do + file=$(echo "${line}" | cut -f1 -d:) + realline=$(echo "${line}" | cut -f2 -d:) + text=$(echo "${line}" | cut -f3- -d:) + idxline="${file}:${realline}:" + uniline=$(${GREP} "${idxline}" "${GITUNIDIFFLINES}" | cut -f3 -d: ) - if [[ ${#JIRA_TEST_TABLE[@]} -gt 0 ]]; then - seccoladj=$(findlargest 1 "${JIRA_TEST_TABLE[@]}") - printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason" - i=0 - until [[ $i -eq ${#JIRA_TEST_TABLE[@]} ]]; do - ourstring=$(echo "${JIRA_TEST_TABLE[${i}]}" | tr -s ' ') - vote=$(echo "${ourstring}" | cut -f2 -d\|) - subs=$(echo "${ourstring}" | cut -f3 -d\|) - printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}" - ((i=i+1)) + for bugs in ${BUGSYSTEMS}; do + if declare -f ${bugs}_linecomments >/dev/null;then + "${bugs}_linecomments" "${title}" "${file}" "${realline}" "${uniline}" "${text}" + fi done - fi - - printf "\n\n|| Subsystem || Report/Notes ||\n" - echo "============================================================================" - i=0 - - until [[ $i -eq ${#JIRA_FOOTER_TABLE[@]} ]]; do - comment=$(echo "${JIRA_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") - printf "%s\n" "${comment}" - ((i=i+1)) - done + done < "${fn}" } -## @description Print out the finished details to the JIRA issue +## @description Write the final output to the selected bug system ## @audience private ## @stability evolving ## @replaceable no -## @param runresult -function output_to_jira +function bugsystem_finalreport { - local result=$1 - local i - local commentfile=${PATCH_DIR}/commentfile - local comment - - rm "${commentfile}" 2>/dev/null - - if [[ ${JENKINS} != "true" ]] ; then - return 0 - fi - - big_console_header "Adding comment to JIRA" - - add_jira_footer "Console output" "${BUILD_URL}console" - - if [[ ${result} == 0 ]]; then - add_jira_header "(/) *{color:green}+1 overall{color}*" - else - add_jira_header "(x) *{color:red}-1 overall{color}*" - fi - - - { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" - - i=0 - until [[ $i -eq ${#JIRA_HEADER[@]} ]]; do - printf "%s\n" "${JIRA_HEADER[${i}]}" >> "${commentfile}" - ((i=i+1)) - done - - { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" - - echo "|| Vote || Subsystem || Runtime || Comment ||" >> "${commentfile}" - - i=0 - until [[ $i -eq ${#JIRA_COMMENT_TABLE[@]} ]]; do - printf "%s\n" "${JIRA_COMMENT_TABLE[${i}]}" >> "${commentfile}" - ((i=i+1)) - done - - - if [[ ${#JIRA_TEST_TABLE[@]} -gt 0 ]]; then - { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" - - echo "|| Reason || Tests ||" >> "${commentfile}" - i=0 - until [[ $i -eq ${#JIRA_TEST_TABLE[@]} ]]; do - printf "%s\n" "${JIRA_TEST_TABLE[${i}]}" >> "${commentfile}" - ((i=i+1)) - done - fi + declare bugs - { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}" - - echo "|| Subsystem || Report/Notes ||" >> "${commentfile}" - i=0 - until [[ $i -eq ${#JIRA_FOOTER_TABLE[@]} ]]; do - comment=$(echo "${JIRA_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${BUILD_URL}artifact/patchprocess,g") - printf "%s\n" "${comment}" >> "${commentfile}" - ((i=i+1)) + for bugs in ${BUGSYSTEMS}; do + if declare -f ${bugs}_finalreport >/dev/null;then + "${bugs}_finalreport" "${@}" + fi done - - printf "\n\nThis message was automatically generated.\n\n" >> "${commentfile}" - - write_to_jira "${commentfile}" } ## @description Clean the filesystem as appropriate and then exit @@ -2517,9 +2725,9 @@ function cleanup_and_exit # there is no need to move it since we assume that # Jenkins or whatever already knows where it is at # since it told us to put it there! - relative_patchdir >/dev/null + relative_dir "${PATCH_DIR}" >/dev/null if [[ $? == 1 ]]; then - hadoop_debug "mv ${PATCH_DIR} ${BASEDIR}" + yetus_debug "mv ${PATCH_DIR} ${BASEDIR}" mv "${PATCH_DIR}" "${BASEDIR}" fi fi @@ -2538,17 +2746,15 @@ function postcheckout local routine local plugin - for routine in find_java_home verify_patch_file - do + for routine in find_java_home verify_patch_file; do verify_patchdir_still_exists - hadoop_debug "Running ${routine}" + yetus_debug "Running ${routine}" ${routine} (( RESULT = RESULT + $? )) if [[ ${RESULT} != 0 ]] ; then - output_to_console 1 - output_to_jira 1 + bugsystem_finalreport 1 cleanup_and_exit 1 fi done @@ -2558,14 +2764,13 @@ function postcheckout if declare -f ${plugin}_postcheckout >/dev/null 2>&1; then - hadoop_debug "Running ${plugin}_postcheckout" + yetus_debug "Running ${plugin}_postcheckout" #shellcheck disable=SC2086 ${plugin}_postcheckout (( RESULT = RESULT + $? )) if [[ ${RESULT} != 0 ]] ; then - output_to_console 1 - output_to_jira 1 + bugsystem_finalreport 1 cleanup_and_exit 1 fi fi @@ -2586,7 +2791,7 @@ function preapply do verify_patchdir_still_exists - hadoop_debug "Running ${routine}" + yetus_debug "Running ${routine}" ${routine} (( RESULT = RESULT + $? )) @@ -2597,7 +2802,7 @@ function preapply if declare -f ${plugin}_preapply >/dev/null 2>&1; then - hadoop_debug "Running ${plugin}_preapply" + yetus_debug "Running ${plugin}_preapply" #shellcheck disable=SC2086 ${plugin}_preapply @@ -2616,32 +2821,21 @@ function postapply local plugin local retval - compute_gitdiff "${GITDIFFLINES}" + compute_gitdiff - check_javac + check_patch_javac retval=$? if [[ ${retval} -gt 1 ]] ; then - output_to_console 1 - output_to_jira 1 + bugsystem_finalreport 1 cleanup_and_exit 1 fi ((RESULT = RESULT + retval)) - for routine in check_javadoc check_apachelicense check_site - do - verify_patchdir_still_exists - hadoop_debug "Running ${routine}" - $routine - - (( RESULT = RESULT + $? )) - - done - for plugin in ${PLUGINS}; do verify_patchdir_still_exists if declare -f ${plugin}_postapply >/dev/null 2>&1; then - hadoop_debug "Running ${plugin}_postapply" + yetus_debug "Running ${plugin}_postapply" #shellcheck disable=SC2086 ${plugin}_postapply (( RESULT = RESULT + $? )) @@ -2658,10 +2852,16 @@ function postinstall local routine local plugin - for routine in check_mvn_eclipse check_findbugs + if declare -f ${BUILDTOOL}_postapply_install >/dev/null; then + "${BUILDTOOL}_postapply_install" + (( RESULT = RESULT + $? )) + fi + + verify_patchdir_still_exists + for routine in check_patch_javadoc do verify_patchdir_still_exists - hadoop_debug "Running ${routine}" + yetus_debug "Running ${routine}" ${routine} (( RESULT = RESULT + $? )) done @@ -2669,13 +2869,12 @@ function postinstall for plugin in ${PLUGINS}; do verify_patchdir_still_exists if declare -f ${plugin}_postinstall >/dev/null 2>&1; then - hadoop_debug "Running ${plugin}_postinstall" + yetus_debug "Running ${plugin}_postinstall" #shellcheck disable=SC2086 ${plugin}_postinstall (( RESULT = RESULT + $? )) fi done - } ## @description Driver to execute _tests routines @@ -2698,7 +2897,7 @@ function runtests for plugin in ${PLUGINS}; do verify_patchdir_still_exists if declare -f ${plugin}_tests >/dev/null 2>&1; then - hadoop_debug "Running ${plugin}_tests" + yetus_debug "Running ${plugin}_tests" #shellcheck disable=SC2086 ${plugin}_tests (( RESULT = RESULT + $? )) @@ -2723,13 +2922,49 @@ function importplugins fi if [[ -n "${USER_PLUGIN_DIR}" && -d "${USER_PLUGIN_DIR}" ]]; then - hadoop_debug "Loading user provided plugins from ${USER_PLUGIN_DIR}" + yetus_debug "Loading user provided plugins from ${USER_PLUGIN_DIR}" files=("${files[@]}" ${USER_PLUGIN_DIR}/*.sh) fi for i in "${files[@]}"; do - hadoop_debug "Importing ${i}" - . "${i}" + if [[ -f ${i} ]]; then + yetus_debug "Importing ${i}" + . "${i}" + fi + done + + if [[ -z ${PERSONALITY} + && -f "${BINDIR}/personality/${PROJECT_NAME}.sh" ]]; then + PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" + fi + + if [[ -n ${PERSONALITY} ]]; then + if [[ ! -f ${PERSONALITY} ]]; then + if [[ -f "${BINDIR}/personality/${PROJECT_NAME}.sh" ]]; then + PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" + else + yetus_debug "Can't find ${PERSONALITY} to import." + return + fi + fi + yetus_debug "Importing ${PERSONALITY}" + . "${PERSONALITY}" + fi +} + +## @description Let plugins also get a copy of the arguments +## @audience private +## @stability evolving +## @replaceable no +function parse_args_plugins +{ + for plugin in ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS}; do + if declare -f ${plugin}_parse_args >/dev/null 2>&1; then + yetus_debug "Running ${plugin}_parse_args" + #shellcheck disable=SC2086 + ${plugin}_parse_args "$@" + (( RESULT = RESULT + $? )) + fi done } @@ -2742,6 +2977,71 @@ function add_plugin PLUGINS="${PLUGINS} $1" } +## @description Register test-patch.d bugsystems +## @audience public +## @stability stable +## @replaceable no +function add_bugsystem +{ + BUGSYSTEMS="${BUGSYSTEMS} $1" +} + +## @description Register test-patch.d test output formats +## @audience public +## @stability stable +## @replaceable no +function add_test_format +{ + TESTFORMATS="${TESTFORMATS} $1" +} + +## @description Register test-patch.d build tools +## @audience public +## @stability stable +## @replaceable no +function add_build_tool +{ + BUILDTOOLS="${BUILDTOOLS} $1" +} + +## @description Calculate the differences between the specified files +## @description and output it to stdout. +## @audience public +## @stability evolving +## @replaceable no +function calcdiffs +{ + local orig=$1 + local new=$2 + local tmp=${PATCH_DIR}/pl.$$.${RANDOM} + local count=0 + local j + + # first, pull out just the errors + # shellcheck disable=SC2016 + ${AWK} -F: '{print $NF}' "${orig}" > "${tmp}.branch" + + # shellcheck disable=SC2016 + ${AWK} -F: '{print $NF}' "${new}" > "${tmp}.patch" + + # compare the errors, generating a string of line + # numbers. Sorry portability: GNU diff makes this too easy + ${DIFF} --unchanged-line-format="" \ + --old-line-format="" \ + --new-line-format="%dn " \ + "${tmp}.branch" \ + "${tmp}.patch" > "${tmp}.lined" + + # now, pull out those lines of the raw output + # shellcheck disable=SC2013 + for j in $(cat "${tmp}.lined"); do + # shellcheck disable=SC2086 + head -${j} "${new}" | tail -1 + done + + rm "${tmp}.branch" "${tmp}.patch" "${tmp}.lined" 2>/dev/null +} + ############################################################################### ############################################################################### ############################################################################### @@ -2754,11 +3054,11 @@ parse_args "$@" importplugins -locate_patch +parse_args_plugins "$@" -find_changed_files +finish_docker_stats -determine_needed_tests +locate_patch # from here on out, we'll be in ${BASEDIR} for cwd # routines need to pushd/popd if they change. @@ -2770,28 +3070,39 @@ if [[ ${JENKINS} == "true" ]] ; then fi fi +find_changed_files + check_reexec +determine_needed_tests + postcheckout +fullyqualifyjdks + +prepopulate_footer + find_changed_modules preapply apply_patch_file -postapply +# we find changed modules again +# in case the patch adds or removes a module +# this also means that test suites need to be +# aware that there might not be a 'before' +find_changed_modules -check_mvn_install +postapply postinstall runtests -close_jira_footer +finish_vote_table -close_jira_table +finish_footer_table -output_to_console ${RESULT} -output_to_jira ${RESULT} +bugsystem_finalreport ${RESULT} cleanup_and_exit ${RESULT}