H12257

dev-support/docs/README.md

@@ -0,0 +1,30 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Overview
+
+Yetus helps community driven software projects improve their contribution and release processes by providing:
+
+* A robust system for automatically checking new contributions against a variety of community accepted requirements
+* The means to document a well defined supported interface for downstream projects
+* Tooling to help release managers generate release documentation based on the information provided by community issue trackers and source repositories
+
+# Yetus Precommit
+
+The Yetus Precommit Patch Tester allows projects to codify their patch acceptance criteria and then evaluate incoming contributions prior to review by a committer.
+
+* Take a quick look at [our glossary of terms](precommit-glossary.md) to ensure you are familiar with the ASF and Maven jargon we'll use as terminology specific to this project.
+* For an overview of Yetus' philosophy on testing contributions and how evaluation is performed, see our [overview](precommit-architecture.md).
+* To get started on your project, including an explanation of what we'll expect in a runtime environment and what optional utilities we'll leverage, read through the [basic usage guide](precommit-basic.md).
+* If your project has advanced requirements such as module relationships not expressed in Maven, special profiles, or a need on os-specific prerequisites not managed by Maven then you'll need to use our [advanced usage guide](precommit-advanced.md).

dev-support/docs/precommit-advanced.md

@@ -0,0 +1,243 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+test-patch
+==========
+
+* [Docker Support](#Docker_Support)
+* [Maven Specific](#Maven_Specific)
+* [Ant Specific](#Ant_Specific)
+* [Plug-ins](#Plug-ins)
+* [Configuring for Other Projects](#Configuring_for_Other_Projects)
+* [Important Variables](#Important_Variables)
+
+# Docker Support
+
+By default, test-patch runs in the same shell where it was launched.  It can alternatively use Docker to launch itself into a container.  This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java.
+
+The `--docker` parameter tells test-patch to run in Docker mode. The `--dockerfile` parameter allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary binaries and tooling needed to run the test.  However be aware that test-patch will copy this file and append its necessary hooks to re-launch itself prior to executing docker.
+
+NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX) or C:\Users (Windows) as the base and patchprocess directories (specified by the --basedir and --patch-dir options respectively), because automatically mountable directories are limited to them. See [the Docker documentation](https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume).
+
+Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --jenkins mode.  In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system.
+
+# Maven Specific
+
+## Command Arguments
+
+test-patch always passes --batch-mode to maven to force it into non-interactive mode.  Additionally, some tests will also force -fae in order to get all of messages/errors during that mode.  It *does not* pass -DskipTests.  Additional arguments should be handled via the personality.
+
+## Test Profile
+
+By default, test-patch will pass -Ptest-patch to Maven. This will allow you to configure special actions that should only happen when running underneath test-patch.
+
+# Ant Specific
+
+## Command Arguments
+
+test-patch always passes -noinput to Ant.  This force ant to be non-interactive.
+
+# Plug-ins
+
+test-patch allows one to add to its basic feature set via plug-ins.  There is a directory called test-patch.d off of the directory where test-patch.sh lives.  Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary.
+
+## Test Plug-ins
+
+Every test plugin must have one line in order to be recognized:
+
+```bash
+add_plugin <pluginname>
+```
+
+This function call registers the `pluginname` so that test-patch knows that it exists.  This plug-in name also acts as the key to the custom functions that you can define. For example:
+
+```bash
+function pluginname_filefilter
+```
+
+This function gets called for every file that a patch may contain.  This allows the plug-in author to determine if this plug-in should be called, what files it might need to analyze, etc.
+
+Similarly, there are other functions that may be defined during the test-patch run:
+
+* pluginname_postcheckout
+    - executed prior to the patch being applied but after the git repository is setup.  This is useful for any early error checking that might need to be done before any heavier work.
+
+* pluginname_preapply
+    - executed prior to the patch being applied.  This is useful for any "before"-type data collection for later comparisons.
+
+* pluginname_postapply
+    - executed after the patch has been applied.  This is useful for any "after"-type data collection.
+
+* pluginname_postinstall
+    - executed after the mvn install test has been done.  If any tests require the Maven repository to be up-to-date with the contents of the patch, this is the place.
+
+* pluginname_tests
+    - executed after the unit tests have completed.
+
+If the plug-in has some specific options, one can use following functions:
+
+* pluginname_usage
+
+    - executed when the help message is displayed. This is used to display the plug-in specific options for the user.
+
+* pluginname_parse_args
+
+    - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+
+    HINT: It is recommended to make the pluginname relatively small, 10 characters at the most.  Otherwise, the ASCII output table may be skewed.
+
+## Bug System Plug-ins
+
+Similar to tests, the ability to add support for bug tracking systems is also handled via a plug-in mechanism.
+
+* pluginname_usage
+
+    - executed when the help message is displayed. This is used to display the plug-in specific options for the user.
+
+* pluginname\_parse\_args
+
+    - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+
+
+* pluginname\_locate\_patch
+
+    - Given input from the user, download the patch if possible.
+
+* pluginname\_determine\_branch
+
+    - Using any heuristics available, return the branch to process, if possible.
+
+* pluginname\_determine\_issue
+
+    - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible.  This is typically used to fill in supplementary information in the final output table.
+
+* pluginname_writecomment
+
+    - Given text input, write this output to the bug system as a comment.  NOTE: It is the bug system's responsibility to format appropriately.
+
+* pluginname\_linecomments
+
+    - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
+
+* pluginname_finalreport
+
+    - Write the final result table to the bug system.
+
+# Configuring for Other Projects
+
+It is impossible for any general framework to be predictive about what types of special rules any given project may have, especially when it comes to ordering and Maven profiles.  In order to direct test-patch to do the correct action, a project `personality` should be added that enacts these custom rules.
+
+A personality consists of two functions. One that determines which test types to run and another that allows a project to dictate ordering rules, flags, and profiles on a per-module, per-test run.
+
+There can be only **one** of each personality function defined.
+
+## Test Determination
+
+The `personality_file_tests` function determines which tests to turn on based upon the file name.  It is relatively simple.  For example, to turn on a full suite of tests for Java files:
+
+```bash
+function personality_file_tests
+{
+  local filename=$1
+
+  if [[ ${filename} =~ \.java$ ]]; then
+    add_test findbugs
+    add_test javac
+    add_test javadoc
+    add_test mvninstall
+    add_test unit
+  fi
+
+}
+```
+
+The `add_test` function is used to activate the standard tests.  Additional plug-ins (such as checkstyle), will get queried on their own.
+
+## Module & Profile Determination
+
+Once the tests are determined, it is now time to pick which [modules](precommit-glossary.md#genericoutside-definitions) should get used.  That's the job of the `personality_modules` function.
+
+```bash
+function personality_modules
+{
+
+    clear_personality_queue
+
+...
+
+    personality_enqueue_module <module> <flags>
+
+}
+```
+
+It takes exactly two parameters `repostatus` and `testtype`.
+
+The `repostatus` parameter tells the `personality` function exactly what state the source repository is in.  It can only be in one of two states:  `branch` or `patch`.  `branch` means the patch has not been applied.  The `patch` state is after the patch has been applied.
+
+The `testtype` state tells the personality exactly which test is about to be executed.
+
+In order to communicate back to test-patch, there are two functions for the personality to use.
+
+The first is `clear_personality_queue`. This removes the previous test's configuration so that a new module queue may be built. Custom personality_modules will almost always want to do this as the first action.
+
+The second is `personality_enqueue_module`.  This function takes two parameters.  The first parameter is the name of the module to add to this test's queue.  The second parameter is an option list of additional flags to pass to Maven when processing it. `personality_enqueue_module` may be called as many times as necessary for your project.
+
+    NOTE: A module name of . signifies the root of the repository.
+
+For example, let's say your project uses a special configuration to skip unit tests (-DskipTests).  Running unit tests during a javadoc build isn't very useful and wastes a lot of time. We can write a simple personality check to disable the unit tests:
+
+
+```bash
+function personality_modules
+{
+    local repostatus=$1
+    local testtype=$2
+
+    if [[ ${testtype} == 'javadoc' ]]; then
+        personality_enqueue_module . -DskipTests
+        return
+    fi
+    ...
+
+```
+
+This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool.
+
+
+
+# Important Variables
+
+There are a handful of extremely important system variables that make life easier for personality and plug-in writers.  Other variables may be provided by individual plug-ins.  Check their development documentation for more information.
+
+* BUILD\_NATIVE will be set to true if the system has requested that non-JVM-based code be built (e.g., JNI or other compiled C code). Under Jenkins, this is always true.
+
+* BUILDTOOL specifies which tool is currently being used to drive compilation.  Additionally, many build tools define xyz\_ARGS to pass on to the build tool command line. (e.g., MAVEN\_ARGS if maven is in use).  Projects may set this in their personality.  NOTE: today, only one build tool at a time is supported.  This may change in the future.
+
+* CHANGED\_FILES is a list of all files that appear to be added, deleted, or modified in the patch.
+
+* CHANGED\_UNFILTERED\_MODULES is a list of all modules that house all of the CHANGED\_FILES.  Be aware that the root of the source tree is reported as '.'.
+
+* CHANGED\_MODULES reports which modules that appear to have source code in them.
+
+* HOW\_TO\_CONTRIBUTE should be a URL that points to a project's on-boarding documentation for new users. Currently, it is used to suggest a review of patch naming guidelines. Since this should be project specific information, it is useful to set in a project's personality.
+
+* JIRA\_ISSUE\_RE is to help test-patch when talking to JIRA.  It helps determine if the given project is appropriate for the given JIRA issue.  There are similar variables for GITHUB.
+
+* MODULE and other MODULE\_\* are arrays that contain which modules, the status, etc, to be operated upon. These should be treated as read-only by plug-ins.
+
+* PATCH\_BRANCH\_DEFAULT is the name of the branch in the git repo that is considered the master.  This is useful to set in personalities.
+
+* PATCH\_DIR is the name of the temporary directory that houses test-patch artifacts (such as logs and the patch file itself)
+
+* TEST\_PARALLEL if parallel unit tests have been requested. Project personalities are responsible for actually enabling or ignoring the request. TEST\_THREADS is the number of threads that have been requested to run in parallel.

dev-support/docs/precommit-architecture.md

@@ -0,0 +1,97 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Some Philosophy
+
+* Everyone's time is valuable.  The quicker contributors can get feedback and iterate, the more likely and faster their contribution will get checked in.  A committer should be able to focus on the core issues of a contribution rather than details that can be determined automatically.
+
+* Precommit checks should be fast.  There is no value in testing parts of the source tree that are not immediately impacted by a change.  Unit testing is the target. They are not a replacement for full builds or integration tests.
+
+* Many open source projects have a desire to have this capability.  Why not generalize a solution?
+
+* In many build systems (especially with maven), a modular design has been picked.  Why not leverage that design to make checks faster?
+
+* Projects that use the same language will, with a high degree of certainty, benefit from the same types of checks.
+
+* Portability matters.  Tooling should be as operating system and language agnostic as possible.
+
+# Phases
+
+test-patch works effectively under several different phases:
+
+## Setup
+
+This is where test-patch configures and validates the environment.  Some things done in this phase:
+
+* Defaults
+* Parameter handling
+* Importing plugins and personalities
+* Docker container launching
+* Re-exec support
+* Patch file downloading
+* git repository management (fresh pull, branch switching, etc)
+
+## Post-checkout
+
+Checks done here are *fatal*.
+
+This acts as a verification of all of the setup parts and is the final place to short-cut the full test cycle.  The most significant built-in check done here is verifying the patch file is a valid.
+
+## Pre-apply
+
+This is where the 'before' work is handled.  Some things that typically get checked in this phase:
+
+* The first pass of files and modules that will get patched
+* Validation and information gathering of the source tree pre-patch
+* Author checks
+* Check for modified unit tests
+
+## Patch is Applied
+
+The patch gets applied.  Then a second pass to determine which modules and files have been changed in order to handle any modules that might have added or moved.
+
+## Post-apply
+
+Now that the patch has been applied, many of the same checks performed in the Pre-apply step are done again to build an 'after' picture.
+
+## Post-install
+
+Some tests only work correctly when the repo is up-to-date. So
+mvn install is run to update the local repo and we enter this phase.  Some example tests performed here:
+
+* javadoc
+* Findbugs
+* Maven eclipse integration still works
+
+## Unit Tests
+
+Since unit tests are generally the slowest part of the precommit process, they are run last.  At this point, all the prerequisites to running them should be in place and ready to go.
+
+## Reporting
+
+Finally, the results are reported to the screen and, optionally, to JIRA and/or whatever bug system has been configured.
+
+# Test Flow
+
+The basic workflow for many of the sub-items in individual phases are:
+
+1. print a header, so the end user knows that something is happening
+1. verify if the test is needed.  If so, continue on.  Otherwise, return success and let the next part of the phase execute.
+1. Ask the personality about what modules and what flags should get used
+1. Execute maven (or some other build tool) in the given modules with the given flags. Log the output and record the time and result code.
+1. Do any extra work as appropriate (diffs, counts, etc) and either accept the status and message given by the maven run or change the vote, message, log file, etc, based upon this extra work.
+1. Add the outcome(s) to the report generator
+
+As one can see, the modules list is one of the key inputs into what actually gets executed.  As a result, projects must full flexibility in either adding, modifying, or even removing modules from the test list.  If a personality removes the entire list of modules, then that test should just be ignored.
+