From 4a48aacd6b68dd0a8c12a4a536c29ac0e9acb583 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Thu, 26 Apr 2018 17:23:35 -0400 Subject: [PATCH 1/6] Add a quick tour of the project to CONTRIBUTING Adds a description of the most important subdirectories to `CONTRIBUTING.md` to help folks that are less familiar with the project get their bearings. It reflects that state of the project right now so it will inevitably go out of date. But I'll try and keep it up to date. --- CONTRIBUTING.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 03b2674a4cc8c..a35ad5f78a526 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -209,6 +209,72 @@ Before submitting your changes, run the test suite to make sure that nothing is ./gradlew check ``` +### Project layout + +This repository is split into many top level directories. The most important +ones are: +* docs: Documentation for the project. +* distribution: Builds our tar and zip archives and our rpm and deb packages. +* libs: Libraries used to build other parts of the project. These are meant to + be internal rather than general purpose. We have no plans to + [semver](https://semver.org/) their APIs or accept feature requests + for them. We publish them to maven central because they are + dependencies of our plugin test framework, high level rest client, and + jdbc driver but. +* modules: Features that are shipped with Elasticsearch by default but are not + built in to the server. We typically separate features from the server + because they require permissions that we don't believe *all* of + Elasticsearch should have or because they depend on libraries that we + don't believe *all* of Elasticsearch should depend on. For example, + reindex requires the `connect` permission so it can perform + reindex-from-remote but we don't believe that the *all* of + Elasticsearch should have the "connect". For another example, Painless + is implemented using antlr4 and asm and we don't believe that *all* + of Elasticsearch should have access to them. +* plugins: Officially supported plugins to Elasticsearch. We decide that a + feature should be a plugin rather than built in to the Elasticsearch + server or shipped as a module because we feel that it is only important + to a subset of users, especially if it requires extra dependencies. The + canonical example of this is the ICU analysis plugin. It is important + for folks who want the fairly language neutral ICU analyzer but the + library to implement the analyzer is 11MB. +* qa: Honestly this is kind of in flux and we're not 100% sure where we'll end + up. Right now the directory contains + * Tests that require multiple modules or plugins to work + * Tests that form a cluster made up of multiple versions of Elasticsearch + like full cluster restart, rolling restarts, and mixed version + tests + * Tests that test the Elasticsearch clients in "interesting" places like + the `wildfly` project. + * Tests that test Elasticsearch in funny configurations like with ingest + disabled + * Tests that need to do strange things like install plugins that thrown + uncaught `Throwable`s or add a shutdown hook + But we're not convinced that all of these things *belong* in the qa + directory. We're fairly sure that "tests that require multiple modules + or plugins to work" should just pick a "home" plugin. We're fairly + sure that the multi-version tests *do* belong in qa. Beyond that, we're + not sure. If you want to add a new thing qa module, ask around and + you'll get several opinions. +* server: The server component of Elasticsearch that contains all of the + modules and plugins. Right now things like the high level rest client + depend on the server but we'd like to fix that in the future. +* test: Our test framework and test fixtures. We use the test framework for + testing the server, the plugins, and modules, and pretty much + everything else. We publish the test framework so folks who develope + Elasticsearch plugins can use it to test the plugins. The test fixtures + are external processes that we start before running specific tests + that rely on them. For example, we have an hdfs test that uses + mini-hdfs to test our repository-hdfs plugin. +* x-pack: Commercially licensed code that integrates with the rest of + Elasticsearch. The `docs` subdirectory functions just like the top + level `docs` subdirectory and the `qa` subdirectory functions just like + the top level `qa` subdirectory. The `plugin` subdirectory contains + the x-pack module which runs inside the Elasticsearch process. The + `transport-client` subdirectory contains extensions to Elasticsearch's + standard transport client to work properly with x-pack. + + Contributing as part of a class ------------------------------- In general Elasticsearch is happy to accept contributions that were created as From f32f950575839f488b56a1c0b3ec70ca8c909092 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Fri, 27 Apr 2018 14:03:51 -0400 Subject: [PATCH 2/6] Reorganize --- CONTRIBUTING.md | 147 ++++++++++++++++++++++++++++-------------------- 1 file changed, 85 insertions(+), 62 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a35ad5f78a526..d00d62cb1e091 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,7 +41,7 @@ We enjoy working with contributors to get their code accepted. There are many ap Note that it is unlikely the project will merge refactors for the sake of refactoring. These types of pull requests have a high cost to maintainers in reviewing and testing with little to no tangible benefit. This especially includes changes generated by tools. For example, -converting all generic interface instances to use the diamond operator. +converting all generic interface instances to use the diamond operator. The process for contributing to any of the [Elastic repositories](https://github.com/elastic/) is similar. Details for individual projects can be found below. @@ -212,67 +212,90 @@ Before submitting your changes, run the test suite to make sure that nothing is ### Project layout This repository is split into many top level directories. The most important -ones are: -* docs: Documentation for the project. -* distribution: Builds our tar and zip archives and our rpm and deb packages. -* libs: Libraries used to build other parts of the project. These are meant to - be internal rather than general purpose. We have no plans to - [semver](https://semver.org/) their APIs or accept feature requests - for them. We publish them to maven central because they are - dependencies of our plugin test framework, high level rest client, and - jdbc driver but. -* modules: Features that are shipped with Elasticsearch by default but are not - built in to the server. We typically separate features from the server - because they require permissions that we don't believe *all* of - Elasticsearch should have or because they depend on libraries that we - don't believe *all* of Elasticsearch should depend on. For example, - reindex requires the `connect` permission so it can perform - reindex-from-remote but we don't believe that the *all* of - Elasticsearch should have the "connect". For another example, Painless - is implemented using antlr4 and asm and we don't believe that *all* - of Elasticsearch should have access to them. -* plugins: Officially supported plugins to Elasticsearch. We decide that a - feature should be a plugin rather than built in to the Elasticsearch - server or shipped as a module because we feel that it is only important - to a subset of users, especially if it requires extra dependencies. The - canonical example of this is the ICU analysis plugin. It is important - for folks who want the fairly language neutral ICU analyzer but the - library to implement the analyzer is 11MB. -* qa: Honestly this is kind of in flux and we're not 100% sure where we'll end - up. Right now the directory contains - * Tests that require multiple modules or plugins to work - * Tests that form a cluster made up of multiple versions of Elasticsearch - like full cluster restart, rolling restarts, and mixed version - tests - * Tests that test the Elasticsearch clients in "interesting" places like - the `wildfly` project. - * Tests that test Elasticsearch in funny configurations like with ingest - disabled - * Tests that need to do strange things like install plugins that thrown - uncaught `Throwable`s or add a shutdown hook - But we're not convinced that all of these things *belong* in the qa - directory. We're fairly sure that "tests that require multiple modules - or plugins to work" should just pick a "home" plugin. We're fairly - sure that the multi-version tests *do* belong in qa. Beyond that, we're - not sure. If you want to add a new thing qa module, ask around and - you'll get several opinions. -* server: The server component of Elasticsearch that contains all of the - modules and plugins. Right now things like the high level rest client - depend on the server but we'd like to fix that in the future. -* test: Our test framework and test fixtures. We use the test framework for - testing the server, the plugins, and modules, and pretty much - everything else. We publish the test framework so folks who develope - Elasticsearch plugins can use it to test the plugins. The test fixtures - are external processes that we start before running specific tests - that rely on them. For example, we have an hdfs test that uses - mini-hdfs to test our repository-hdfs plugin. -* x-pack: Commercially licensed code that integrates with the rest of - Elasticsearch. The `docs` subdirectory functions just like the top - level `docs` subdirectory and the `qa` subdirectory functions just like - the top level `qa` subdirectory. The `plugin` subdirectory contains - the x-pack module which runs inside the Elasticsearch process. The - `transport-client` subdirectory contains extensions to Elasticsearch's - standard transport client to work properly with x-pack. +ones are + +#### `docs` +Documentation for the project. + +#### `distribution` +Builds our tar and zip archives and our rpm and deb packages. + +#### `libs` +Libraries used to build other parts of the project. These are meant to be +internal rather than general purpose. We have no plans to +[semver](https://semver.org/) their APIs or accept feature requests for them. +We publish them to maven central because they are dependencies of our plugin +test framework, high level rest client, and jdbc driver but they really aren't +general purpose enough to *belong* in maven central. We're still working out +what to do here.... + +#### `modules` +Features that are shipped with Elasticsearch by default but are not built in to +the server. We typically separate features from the server because they require +permissions that we don't believe *all* of Elasticsearch should have or because +they depend on libraries that we don't believe *all* of Elasticsearch should +depend on. + +For example, reindex requires the `connect` permission so it can perform +reindex-from-remote but we don't believe that the *all* of Elasticsearch should +have the "connect". For another example, Painless is implemented using antlr4 +and asm and we don't believe that *all* of Elasticsearch should have access to +them. + +#### `plugins` +Officially supported plugins to Elasticsearch. We decide that a feature should +be a plugin rather than shipped as a module because we feel that it is only +important to a subset of users, especially if it requires extra dependencies. + +The canonical example of this is the ICU analysis plugin. It is important for +folks who want the fairly language neutral ICU analyzer but the library to +implement the analyzer is 11MB so we don't ship it with Elasticsearch by +default. + +Another example is the `discovery-gce` plugin. It is *vital* to folks running +in [GCP](https://cloud.google.com/) but useless otherwise and it depends on a +dozen extra jars. + +#### `qa` +Honestly this is kind of in flux and we're not 100% sure where we'll end up. +Right now the directory contains +* Tests that require multiple modules or plugins to work +* Tests that form a cluster made up of multiple versions of Elasticsearch like + full cluster restart, rolling restarts, and mixed version tests +* Tests that test the Elasticsearch clients in "interesting" places like the + `wildfly` project. +* Tests that test Elasticsearch in funny configurations like with ingest + disabled +* Tests that need to do strange things like install plugins that thrown + uncaught `Throwable`s or add a shutdown hook +But we're not convinced that all of these things *belong* in the qa directory. +We're fairly sure that "tests that require multiple modules or plugins to work" +should just pick a "home" plugin. We're fairly sure that the multi-version +tests *do* belong in qa. Beyond that, we're not sure. If you want to add a new +thing qa module, ask around and you'll get several opinions. + +#### `server` +The server component of Elasticsearch that contains all of the modules and +plugins. Right now things like the high level rest client depend on the server +but we'd like to fix that in the future. + +#### `test` +Our test framework and test fixtures. We use the test framework for testing the +server, the plugins, and modules, and pretty much everything else. We publish +the test framework so folks who develope Elasticsearch plugins can use it to +test the plugins. The test fixtures are external processes that we start before +running specific tests that rely on them. + +For example, we have an hdfs test that uses mini-hdfs to test our +repository-hdfs plugin. + +#### `x-pack` +Commercially licensed code that integrates with the rest of Elasticsearch. The +`docs` subdirectory functions just like the top level `docs` subdirectory and +the `qa` subdirectory functions just like the top level `qa` subdirectory. The +`plugin` subdirectory contains the x-pack module which runs inside the +Elasticsearch process. The `transport-client` subdirectory contains extensions +to Elasticsearch's standard transport client to work properly with x-pack. Contributing as part of a class From c5fcc61c363864df5d7ecdff07cb7701adf06548 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Mon, 30 Apr 2018 11:39:14 -0400 Subject: [PATCH 3/6] Cleanup --- CONTRIBUTING.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d00d62cb1e091..006e334904a06 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -261,18 +261,18 @@ Honestly this is kind of in flux and we're not 100% sure where we'll end up. Right now the directory contains * Tests that require multiple modules or plugins to work * Tests that form a cluster made up of multiple versions of Elasticsearch like - full cluster restart, rolling restarts, and mixed version tests +full cluster restart, rolling restarts, and mixed version tests * Tests that test the Elasticsearch clients in "interesting" places like the - `wildfly` project. +`wildfly` project. * Tests that test Elasticsearch in funny configurations like with ingest - disabled +disabled * Tests that need to do strange things like install plugins that thrown - uncaught `Throwable`s or add a shutdown hook +uncaught `Throwable`s or add a shutdown hook But we're not convinced that all of these things *belong* in the qa directory. -We're fairly sure that "tests that require multiple modules or plugins to work" +We're fairly sure that tests that require multiple modules or plugins to work should just pick a "home" plugin. We're fairly sure that the multi-version tests *do* belong in qa. Beyond that, we're not sure. If you want to add a new -thing qa module, ask around and you'll get several opinions. +qa project, open a PR and be ready to discuss options. #### `server` The server component of Elasticsearch that contains all of the modules and @@ -282,7 +282,7 @@ but we'd like to fix that in the future. #### `test` Our test framework and test fixtures. We use the test framework for testing the server, the plugins, and modules, and pretty much everything else. We publish -the test framework so folks who develope Elasticsearch plugins can use it to +the test framework so folks who develop Elasticsearch plugins can use it to test the plugins. The test fixtures are external processes that we start before running specific tests that rely on them. From baad66d272917dad5e901be07fa7e7907c332719 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Mon, 30 Apr 2018 11:39:47 -0400 Subject: [PATCH 4/6] : --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 006e334904a06..334411172529e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -212,7 +212,7 @@ Before submitting your changes, run the test suite to make sure that nothing is ### Project layout This repository is split into many top level directories. The most important -ones are +ones are: #### `docs` Documentation for the project. @@ -269,7 +269,7 @@ disabled * Tests that need to do strange things like install plugins that thrown uncaught `Throwable`s or add a shutdown hook But we're not convinced that all of these things *belong* in the qa directory. -We're fairly sure that tests that require multiple modules or plugins to work +We're fairly sure that tests that require multiple modules or plugins to work` should just pick a "home" plugin. We're fairly sure that the multi-version tests *do* belong in qa. Beyond that, we're not sure. If you want to add a new qa project, open a PR and be ready to discuss options. From 3de0f14835541f3d84957ea1a39608d113da82c2 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Mon, 30 Apr 2018 11:40:45 -0400 Subject: [PATCH 5/6] less dots --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 334411172529e..2750335c29383 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -227,7 +227,7 @@ internal rather than general purpose. We have no plans to We publish them to maven central because they are dependencies of our plugin test framework, high level rest client, and jdbc driver but they really aren't general purpose enough to *belong* in maven central. We're still working out -what to do here.... +what to do here. #### `modules` Features that are shipped with Elasticsearch by default but are not built in to From 6e8a4de24cf8647651c664a1e1cbc4db217f7687 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Mon, 7 May 2018 17:00:44 -0400 Subject: [PATCH 6/6] Sneaky ` --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2750335c29383..f1f78910e9a4d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -269,7 +269,7 @@ disabled * Tests that need to do strange things like install plugins that thrown uncaught `Throwable`s or add a shutdown hook But we're not convinced that all of these things *belong* in the qa directory. -We're fairly sure that tests that require multiple modules or plugins to work` +We're fairly sure that tests that require multiple modules or plugins to work should just pick a "home" plugin. We're fairly sure that the multi-version tests *do* belong in qa. Beyond that, we're not sure. If you want to add a new qa project, open a PR and be ready to discuss options.