Converting DependencyLicensesTask and UpdateShasTask to java #35231

Megamiun · 2018-11-04T02:53:56Z

Relates to #34459

alpar-t · 2018-11-05T11:57:58Z

elasticmachine · 2018-11-05T12:29:58Z

Pinging @elastic/es-core-infra

alpar-t

Thansk for your contribution @Megamiun .
I left some comments. Still have to look at the tests.
Perhaps would be good to add a testkit based test to make sure that the up-to-date checks work as they should. Calling the task twice should result in the second time being up-to-date

buildSrc/build.gradle

buildSrc/src/main/java/org/elasticsearch/gradle/precommit/DependencyLicensesTask.java

alpar-t · 2018-11-06T12:19:50Z

buildSrc/src/main/java/org/elasticsearch/gradle/precommit/DependencyLicensesTask.java

I think we should expose this differently, but that can go into a separate PR.
Assigning this from configurations makes it way too easy to resolve the configurations too early.
We should get the dependencies on project like it's being assigned in BuildPlugin and offer ways to filter them, like having a List<Predicate> for projects that need it, like server/build.gradle

Didn't do this one yet, think I will leave it to another future PR

Megamiun · 2018-11-07T03:38:19Z

About the test-kit tests, I don't understand corretcly what should I do. Just a test to see if the status would be UP-TO-DATE on a repeated run?

alpar-t · 2018-11-07T11:30:50Z

You can look at tests that implement GradleIntegrationTestCase for some examples.
You have to set up a project that applies the build plugin and check that the tasks do what it's expected of them. Then run it a second time and assert that it's up to date. This will catch incorrectly wired task inputs for the most part. You could also change some of the inputs or outputs and validate that it's not considered up to date.

@elasticmachine test this please

alpar-t · 2018-11-07T16:40:43Z

@Megamiun you'll have to merge master to get the builds passing.

Megamiun · 2018-11-26T11:18:48Z

Sorry for the delay, almost finished here.

But I saw something, it seems this task can not be used as up-to-date. I put it to run with --info and received:

Task ':x-pack:plugin:ml:dependencyLicenses' is not up-to-date because:
  Task has not declared any outputs despite executing actions.

It seems that by not existing output, gradle doesn't skip execution because of up-to-date status.

Probably by tomorrow I can make the MR

tomcallahan · 2019-01-25T14:50:27Z

@Megamiun are you still interested in pursuing this PR ?

Megamiun · 2019-01-25T19:37:10Z

Yes, thanks for remembering me, will try to do this weekend.

But even so, @atorok, about the UP-TO-DATE, is it really the desired behaviour?

rjernst · 2019-01-25T20:31:57Z

about the UP-TO-DATE, is it really the desired behaviour?

Nope, we should fix this (although it doesn't have to be in this PR). The way we have done this in other cases is to use a "marker" file that we simply touch when the task completes, and configure that as the task output. For example, see ForbiddenPatternsTask.

…task-to-java

alpar-t

@elasticmachine test this please

buildSrc/src/main/java/org/elasticsearch/gradle/precommit/DependencyLicensesTask.java

alpar-t · 2019-03-06T15:29:06Z

buildSrc/build.gradle

  compile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
-
+
+  compile "commons-codec:commons-codec:${props.getProperty('commonscodec')}"


Why are we adding this would try to avoid increasing the transitive dependencies if possible

I added this to add the possibility of encoding the sha1 byte array to an hex string.

It seems the way to do it is a little more laborious in pure java and the encodeHex method was a Groovy method.

If prefered, I can change the impl to use java directly.

And when making changes, I just need to rebase my branch, correct?

Oh, and the integTest task takes some hours(it didn't complete yet), on my computer, is it normal?

@Megamiun thanks for the explanation, I think that's good enough reason to add it.
Unfortunately our integ tests do take that long currently, there's many of them and we are working to optimize execution to make them faster.

@elasticmachine test this please

alpar-t · 2019-03-07T07:52:29Z

@elasticmachine tesdt this please

alpar-t · 2019-03-07T07:55:35Z

@elasticmachine test this please

…task-to-java

alpar-t · 2019-03-07T13:54:12Z

@elasticmachine test this please

Megamiun · 2019-03-08T01:46:39Z

Hello, tried to resolve the IT tests problem, is it possible to run again the tests? Unfortunately the time it takes to tuns the tests makes it not possible to test it on my pc

alpar-t · 2019-03-08T06:56:59Z

@Megamiun you can run the test you added alone with this command: ./gradlew :build-tools:integTest --tests="*DependencyLicensesTasksIT"

@elasticmachine test this please

Megamiun · 2019-03-09T03:18:10Z

I think maybe there is a problem with my pc, it has been eleven hours and the integtests are still running. And I used the command to run only mine.

I analysed the jarHell its and I tried to make some changes that to me seems to resolve the problem

Megamiun · 2019-03-19T03:20:05Z

Hello, @atorok?

alpar-t · 2019-03-19T06:00:23Z

buildSrc/src/test/java/org/elasticsearch/gradle/precommit/DependencyLicensesTasksIT.java

        GradleRunner runner = getRunner("dependencies_missing");

-        assertTaskFailed(runner.build(), ":dependencies_missing:dependencyLicenses");
+        assertTaskFailed(runner.buildAndFail(), ":dependencies_missing:dependencyLicenses");


This change is needed, but when asserting for failure we should assert for something in the output to make sure we are getting the failure we want.
I can see that from the last test run we got Plugin with id 'elasticsearch.build' not found. so I don't think we are getting the intended failure here yet the test would pass.

The reason for the failure is that you must use the new Gradle syntax to apply the plugin. take a look at buildSrc/src/testKit/elasticsearch.build/build.gradle

Megamiun · 2019-05-06T23:32:59Z

Sorry, @atorok, I closed this PR by error. Now I am doing one more try of running the tests. I will open a new PR once I finish this.

I found a way to run the integTests locally

alpar-t self-requested a review November 5, 2018 11:58

alpar-t mentioned this pull request Nov 5, 2018

Replace groovy build-tools (buildSrc) with java #34459

Open

31 tasks

cbuescher added >enhancement :Delivery/Build Build or test infrastructure labels Nov 5, 2018

alpar-t reviewed Nov 6, 2018

View reviewed changes

Megamiun force-pushed the refactor/updateshastask-to-java branch from 5c4f895 to 3c116a9 Compare November 7, 2018 03:35

michaelbaamonde added the team-discuss label Feb 28, 2019

Megamiun force-pushed the refactor/updateshastask-to-java branch from 3c116a9 to 5b119c9 Compare March 4, 2019 21:37

Megamiun added 2 commits March 5, 2019 01:05

Converting DependencyLicensesTask and UpdateShasTask to java

0a255bc

Pushing start of IT

90add72

Megamiun force-pushed the refactor/updateshastask-to-java branch from 5b119c9 to 90add72 Compare March 5, 2019 05:56

Correcting tests

33bb6fb

danielmitterdorfer removed the team-discuss label Mar 6, 2019

Merge remote-tracking branch 'origin/master' into refactor/updateshas…

ec17b1b

…task-to-java

alpar-t reviewed Mar 6, 2019

View reviewed changes

alpar-t added 2 commits March 7, 2019 15:42

Merge remote-tracking branch 'origin/master' into refactor/updateshas…

3f80c90

…task-to-java

Fix input annotations

7d24a86

Trying to correct ITs

207e096

Adding plugin classpath

a4f4012

alpar-t reviewed Mar 19, 2019

View reviewed changes

rhamedy mentioned this pull request Apr 25, 2019

Refactor: convert DependenciesInfoTask to Java #41180

Closed

Megamiun closed this May 6, 2019

Megamiun deleted the refactor/updateshastask-to-java branch May 6, 2019 21:25

Megamiun restored the refactor/updateshastask-to-java branch May 6, 2019 21:28

Megamiun mentioned this pull request May 7, 2019

Converting DependencyLicensesTask and UpdateShasTask to java #41921

Merged

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

		compile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"


		compile "commons-codec:commons-codec:${props.getProperty('commonscodec')}"

Converting DependencyLicensesTask and UpdateShasTask to java #35231

Converting DependencyLicensesTask and UpdateShasTask to java #35231

Uh oh!

Conversation

Megamiun commented Nov 4, 2018

Uh oh!

alpar-t commented Nov 5, 2018

Uh oh!

elasticmachine commented Nov 5, 2018

Uh oh!

alpar-t left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alpar-t Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

Megamiun Nov 7, 2018

Choose a reason for hiding this comment

Uh oh!

Megamiun commented Nov 7, 2018

Uh oh!

alpar-t commented Nov 7, 2018

Uh oh!

alpar-t commented Nov 7, 2018

Uh oh!

Megamiun commented Nov 26, 2018

Uh oh!

tomcallahan commented Jan 25, 2019

Uh oh!

Megamiun commented Jan 25, 2019

Uh oh!

rjernst commented Jan 25, 2019

Uh oh!

alpar-t left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alpar-t Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

Megamiun Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

Megamiun Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

alpar-t Mar 7, 2019

Choose a reason for hiding this comment

Uh oh!

alpar-t commented Mar 7, 2019

Uh oh!

alpar-t commented Mar 7, 2019

Uh oh!

alpar-t commented Mar 7, 2019

Uh oh!

Megamiun commented Mar 8, 2019

Uh oh!

alpar-t commented Mar 8, 2019

Uh oh!

Megamiun commented Mar 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Megamiun commented Mar 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alpar-t Mar 19, 2019

Choose a reason for hiding this comment

Uh oh!

Megamiun commented May 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Megamiun commented Mar 9, 2019 •

edited

Loading

Megamiun commented Mar 19, 2019 •

edited

Loading

Megamiun commented May 6, 2019 •

edited

Loading