Deprecation info for joda-java migration #41956

pgomulka · 2019-05-08T15:46:59Z

Implement deprecation checks for joda incompatible patterns

relates #42010

elasticmachine · 2019-05-08T15:47:03Z

Pinging @elastic/es-core-infra

...precation/src/test/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecksTests.java

rjernst

A few comments

test/framework/src/main/java/org/elasticsearch/bootstrap/BootstrapForTesting.java

rjernst · 2019-05-08T17:28:05Z

...in/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java

+
+    static Map<String, String> JODA_PATTERNS_DEPRECATIONS = new LinkedHashMap<>();
+    static {
+        JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era becomes 'y'. 'Y' means week-based-year.");


We already have these deprecation messages in Joda, which are logged when the patterns are parsed. This seems to be doubling them? Additionally, I think some of these are too broad, like the check for y, which users wouldn't need to do anything about. Any deprecation messages should be actionable.

Yes - that was just a draft, but next step is to have those messages in one place and use in both Joda & Deprecation API.
for y pattern check I think it would be fair to notify users about this as it changes to u (I missed this on previous commit). I suppose some users would just ignore this check specifically as a difference is very subtle, but we should at least let them know.

Some of the patterns are not actionable though - like C century is no longer present in java.time. Again users should be aware about this, as their pattern will probably no longer work. Action would be to change your business logic?

...in/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java

gwbrown · 2019-05-08T22:40:52Z

This approach seems reasonable to me, the warning it generates is a little verbose but otherwise should be fine (and it's not like we don't already have some very verbose warnings).

As others have noted, there's a fair bit of commented out code, etc. left, but I assume that's why this PR is still marked as a draft/WIP.

pgomulka · 2019-05-09T11:51:12Z

there are a lot of test failing as the deprecation now includes y and Z (and that adds deprecation to headers). I understand @rjernst and @gwbrown concerns that this might be a bit too verbose.
At the same time, since y has a slightly different meaning now in java.time and Z behaves a little bit different I think it would make sens to warn users about this.
@spinscale what is your view on this?

pgomulka · 2019-05-21T16:22:13Z

docs/reference/indices/templates.asciidoc

 --------------------------------------------------
 // CONSOLE
-// TESTSETUP
+// TEST[setup:name]


I need to figure out how to fix Docs tests. At the moment they fail because of warnings

Normally we would update the docs to not use deprecated functionality. Is that possible here?

it would - actually it would be much easier if I just use a new pattern with 8 prefix. But should this be done across all the docs?
That change is to fix the DocsClientYamlTestSuiteIT. The options is to prefix with8, assert about warnings, disable warning checks alltogether?

server/src/main/java/org/elasticsearch/common/joda/Joda.java

server/src/main/java/org/elasticsearch/common/time/FormatNames.java

gwbrown

Left a few comments, mostly minor.

You mentioned wanting to have more Javadoc and tests, which I agree with as well, plus the commented-out code and TODOs left which I didn't address as this is still marked WIP. Otherwise I still don't see any major problems, and it's already much improved from the last time I looked at this. Thanks for doing this!

gwbrown · 2019-05-21T23:29:18Z

server/src/test/java/org/elasticsearch/common/joda/JavaJodaTimeDuellingTests.java

 public class JavaJodaTimeDuellingTests extends ESTestCase {
+    @Override
+    protected boolean enableWarningsCheck() {
+        // disable warning checks as deprecated patterns are used to compare Joda vs Java results (y - year and Z zone offset)


I would update this note to say that these warnings as tested in JodaWarningTests already, so that's why we don't test them here.

Pretend I copy/pasted this onto every iteration of this comment, but I'm not going to because there's a lot of them.

gwbrown · 2019-05-21T23:39:20Z

server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java

 public class RootObjectMapperTests extends ESSingleNodeTestCase {
+    @Override
+    protected boolean enableWarningsCheck() {
+        // disable warning checks as deprecated patterns are used to compare Joda vs Java results (y - year and Z zone offset)


I get doing this for date-specific tests, but there's a lot of classes that have the warnings check disabled here, including some very broad ones. We should be careful about doing this so widely, although I could see an argument that this is okay if it would be a lot of work to use assertWarnings everywhere given these will only be going into 6.8, which should pretty much only be getting bugfixes at this point.

where possible I used explicitly assertWarnings about these patterns, but often it was just not feasible and would require adding lot of them.
and agree, that is only going to be in 6.8 so hopefully would not reduce confidence in these tests

...in/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java

gwbrown · 2019-05-21T23:49:15Z

server/src/main/java/org/elasticsearch/common/joda/JodaDeprecationPatterns.java

+    private static Map<String, String> JODA_PATTERNS_DEPRECATIONS = new LinkedHashMap<>();
+
+    static {
+        JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era becomes 'y'. Use 'Y' for week-based-year.");


I think we should make this a bit clearer as to what we're recommending be done. Something like this:

Suggested change

JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era becomes 'y'. Use 'Y' for week-based-year.");

JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era should be replaced with 'y'. Use 'Y' for week-based-year.");

And similar for the rest of these.

droberts195 · 2019-05-24T10:35:26Z

server/src/main/java/org/elasticsearch/common/joda/JodaDeprecationPatterns.java

+            }
+        }
+        String combinedWarning = warnings.stream()
+                                 .collect(Collectors.joining(";"));


Since this is building a message for the end user it would be nice to put a space after the semi-colon.

good point, this would be more readable with a space

droberts195 · 2019-05-24T10:39:36Z

...precation/src/test/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecksTests.java

+            "https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#breaking_70_java_time_changes",
+            "This index has date fields with deprecated formats: ["+
+                "[type: _doc, field: date_time_field_Y, format: dd-CC||MM-YYYY, " +
+                "suggestion: 'C' century of era is no longer supported.;"+


The line break disguises the fact that there's no space after the semi-colon here. At the moment the message is ...supported.;'Y' year-of-era.... If you accept my other suggestion you'll need to add a space here.

You are right, will add a space after a semi-colon. Should be more readable

pgomulka · 2019-05-24T15:34:30Z

This PR has grown significantly which is sad as I only expected this to be a quick fix.
There are a lot of tests that I had to change, where possible I explicitly asserted about new warnings in a headers, but sometimes I had to turn that check off as it would make this PR just massive.

I have added some more tests to cover the message formatting and different scenarios of the pattern.

@gwbrown @rjernst @droberts195 would you be able to re-review this again?

gwbrown

The deprecation check LGTM, though I left a couple comments. Others may want to weigh in on the rest.

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/section/DoSection.java

...in/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java

…k/deprecation/IndexDeprecationChecks.java Co-Authored-By: Gordon Brown <[email protected]>

pgomulka · 2019-05-27T13:43:27Z

added support for deprecation info on pipelines with date and date_index_name processors
Also I disabled only joda deprecation checks by default only. This was spreading far too much and was not really giving much more confidence as assertions we usually about the same pattern.

gwbrown

Left a couple minor comments but otherwise LGTM. No need for another round of review after addressing these unless you make other substantial changes.

gwbrown · 2019-05-28T21:41:04Z

server/src/test/java/org/elasticsearch/common/joda/JavaJodaTimeDuellingTests.java

+
+        //in joda 'Z' was able to parse 'Z' zulu but in java it fails. You have to use 'X' to do that.
+        //Caused by: java.time.format.DateTimeParseException: Text '2019-01-01T01:01:01.001Z' could not be parsed at index 23
+        //assertSameMillis("2019-01-01T01:01:01.001Z","YYYY-MM-dd'T'HH:mm:ss.SSSZ","8yyyy-MM-dd'T'HH:mm:ss.SSSZ");


I think these lines might be left over from something? Otherwise I'm not sure why they're here.

This whole test case testIncompatiblePatterns was meant to use examples of patterns that are incompatible between joda-java and show how to fix them.
For 'Z' we should go for 'X' so I will just remove this comment.

.../deprecation/src/main/java/org/elasticsearch/xpack/deprecation/ClusterDeprecationChecks.java

…k/deprecation/ClusterDeprecationChecks.java Co-Authored-By: Gordon Brown <[email protected]>

mark-vieira · 2019-05-29T17:25:13Z

FYI, this change is breaking some tests in BWC compatible branches. We'll need to update org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT in 7.1, 7.2 and 7.x.

https://scans.gradle.com/s/z2xy2pasutgty/tests/z3ejz3hl3i73s-wptfavgv7zdvi

pgomulka added 3 commits May 7, 2019 12:31

initial work

8f99fc7

init version

dbc725c

draft working version

937fe06

pgomulka added the :Core/Infra/Core Core issues without another label label May 8, 2019

pgomulka self-assigned this May 8, 2019

droberts195 reviewed May 8, 2019

View reviewed changes

...precation/src/test/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecksTests.java Outdated Show resolved Hide resolved

rjernst reviewed May 8, 2019

View reviewed changes

pgomulka added the WIP label May 8, 2019

pgomulka added 6 commits May 9, 2019 10:17

cleanup

92dd79f

commented code remove

0ab2a6e

joda deprecation keys

353af6a

generic deprecation key

21255e1

empty lines

6e85a05

message fix, prefix8 suggestion

8ca0fee

fixing test for warnings in headers

c94bbe7

pgomulka force-pushed the feature/deprecation_info_joda branch from 9152886 to c94bbe7 Compare May 10, 2019 20:25

pgomulka added 5 commits May 11, 2019 09:18

do not report deprecation when format starts with 8

5e7bdb3

unused improt

ba913f5

remove ignore annotation

7bc3ea7

field extraction

837dea3

changes to support defined format names

23f11f4

pgomulka commented May 21, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/common/joda/Joda.java Show resolved Hide resolved

change imports

a758afb

pgomulka commented May 21, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/common/time/FormatNames.java Show resolved Hide resolved

gwbrown reviewed May 21, 2019

View reviewed changes

compile fix

c282f14

testing combined patterns and removing duplicates

a64c61b

droberts195 reviewed May 24, 2019

View reviewed changes

pgomulka added 4 commits May 24, 2019 12:49

test failures

c853b72

space after ; when combined pattern suggestion

56bc22d

failing tests

c4632c8

unused import

55a233f

gwbrown reviewed May 24, 2019

View reviewed changes

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/section/DoSection.java Outdated Show resolved Hide resolved

...in/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java Outdated Show resolved Hide resolved

pgomulka and others added 7 commits May 25, 2019 19:38

Update x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpac…

9e77047

…k/deprecation/IndexDeprecationChecks.java Co-Authored-By: Gordon Brown <[email protected]>

remove intellij instructions

9b98a79

pipelines support

aa36731

failing tests

3e3c106

checkstyle

f7ee788

deprecation issue message change

eeaf96c

disable joda deprecation checks by default

0dd1ae1

pgomulka added 5 commits May 27, 2019 15:53

remove failing test cases

0afb36d

use constant instead of string for usePrefix8

40509ec

fix tests with warning assertions

7d3de05

remove warning assertion

8b82383

IT test failing fix

85fd30e

gwbrown approved these changes May 28, 2019

View reviewed changes

pgomulka and others added 3 commits May 29, 2019 08:51

Update x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpac…

fe2f17d

…k/deprecation/ClusterDeprecationChecks.java Co-Authored-By: Gordon Brown <[email protected]>

unused comment and message change

edd0aa1

revert resetDeprecationLogger method access

84608c2

pgomulka merged commit 728c5d1 into elastic:6.8 May 29, 2019

jbaiera mentioned this pull request May 29, 2019

Mixed Cluster Tests failing after joda migration #42679

Closed

pgomulka mentioned this pull request Jun 6, 2019

Implement deprecation info API for joda-java migration #42010

Closed

ChrisHegarty unassigned pgomulka Oct 29, 2024

	JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era becomes 'y'. Use 'Y' for week-based-year.");
	JODA_PATTERNS_DEPRECATIONS.put("Y", "'Y' year-of-era should be replaced with 'y'. Use 'Y' for week-based-year.");

Deprecation info for joda-java migration #41956

Deprecation info for joda-java migration #41956

Uh oh!

Conversation

pgomulka commented May 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented May 8, 2019

Uh oh!

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgomulka May 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gwbrown commented May 8, 2019

Uh oh!

pgomulka commented May 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gwbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgomulka commented May 24, 2019

Uh oh!

gwbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pgomulka commented May 27, 2019

Uh oh!

gwbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mark-vieira commented May 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

pgomulka commented May 8, 2019 •

edited

Loading

pgomulka May 9, 2019 •

edited

Loading

pgomulka commented May 9, 2019 •

edited

Loading