Skip to content

SQL: Enhance timestamp escaped literal parsing #52097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Feb 10, 2020
Merged

SQL: Enhance timestamp escaped literal parsing #52097

merged 5 commits into from
Feb 10, 2020

Conversation

@matriv
Copy link
Contributor

@matriv matriv commented Feb 8, 2020 •
edited
Loading

Allow also whitespace (together with T) as a separator between
date and time parts of the timestamp string. E.g.:

{ts '2020-02-08 12.10.45'}

or

{ts '2020-02-08T12.10.45'}

Fixes: #46069

@matriv
SQL: Enhance timestamp escaped literal parsing
3166b8e 
Allow also whitespace ` ` (together with `T`) as a separator between
date and time parts of the timestamp string. E.g.:
```
2020-02-08 12.10.45
```
or
```
2020-02-08T12.10.45
```

Fixes: elastic#46049
@matriv matriv requested review from astefan, bpintea and costin February 8, 2020 15:11
@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 8, 2020

Pinging @elastic/es-search (:Search/SQL)

@matriv
Copy link
Contributor Author

matriv commented Feb 8, 2020

Folks, do you think we could treat it as a bug fix and backport it to previous versions?

costin
costin reviewed Feb 8, 2020
View reviewed changes
x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/util/DateUtils.java Outdated

public static ZonedDateTime ofEscapedLiteral(String dateFormat) {
return ZonedDateTime.parse(dateFormat, DATE_TIME_ESCAPED_LITERAL_FORMATTER.withZone(UTC));
try {
Copy link
Member

@costin costin Feb 8, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be improved by checking whether there's a T separator at its dedicated position (10) in order to select the right parser. This should avoid the fallback in case of an exception.

Copy link
Contributor Author

@matriv matriv Feb 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The position is not dedicated though as the year can be more than 4 digits.
But I can use lastIndexOf('-')

costin
costin approved these changes Feb 8, 2020
View reviewed changes
Copy link
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
P.S. The referred issue seems to be incorrect.

astefan
astefan reviewed Feb 9, 2020
View reviewed changes
Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any other formats weren't supposed to be handled by a more general functionality in which a format could be specified to convert a date to that format?
Like a TO_DATE like function.

@matriv
Copy link
Contributor Author

matriv commented Feb 9, 2020 •
edited
Loading

Any other formats weren't supposed to be handled by a more general functionality in which a format could be specified to convert a date to that format?
Like a TO_DATE like function.

This is just to properly support the ODBC escaped timestamp literals: https://docs.microsoft.com/en-us/sql/odbc/microsoft/jet-date-time-and-timestamp-literals?view=sql-server-ver15

We can still enhance the overall functionality by using introducing a TO_DATE function where the user can specify a custom format.

astefan
astefan reviewed Feb 9, 2020
View reviewed changes
Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments

x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/parser/EscapedFunctionsTests.java Outdated
Comment on lines 85 to 96
boolean hasSecs = randomBoolean();
String secs = "";
if (hasSecs) {
secs = ":55";
}

String fractionalSecs = "";
if (hasSecs) {
fractionalSecs = randomFrom("", ".1", ".12", ".123", ".1234", ".12345", ".123456",
".1234567", ".12345678", ".123456789");
}
return secs + fractionalSecs;
Copy link
Contributor

@astefan astefan Feb 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this whole code could be simplified (and still readable and not cluttered) as:

        if (randomBoolean()) {
           return ":55" + randomFrom("", ".1", ".12", ".123", ".1234", ".12345", ".123456",
                    ".1234567", ".12345678", ".123456789");
        }
        return "";

x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/parser/EscapedFunctionsTests.java
int length = randomIntBetween(4, 9);

if (randomBoolean()) {
sb.append('-');
Copy link
Contributor

@astefan astefan Feb 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this. Why is the date starting with -?

Copy link
Contributor Author

@matriv matriv Feb 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To test negative Year.

@matriv matriv requested a review from astefan February 9, 2020 18:38
astefan
astefan approved these changes Feb 10, 2020
View reviewed changes
Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@matriv matriv merged commit 07c9770 into elastic:master Feb 10, 2020
@matriv matriv deleted the fix-46069 branch February 10, 2020 09:39
matriv added a commit that referenced this pull request Feb 10, 2020
@matriv
SQL: Enhance timestamp escaped literal parsing (#52097)
27265f0 
Allow also whitespace ` ` (together with `T`) as a separator between
date and time parts of the timestamp string. E.g.:
```
{ts '2020-02-08 12.10.45'}
```
or
```
{ts '2020-02-08T12.10.45'}
```

Fixes: #46069
(cherry picked from commit 07c9770)
@codebrain codebrain mentioned this pull request Apr 1, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@costin costin costin approved these changes

@astefan astefan astefan approved these changes

@bpintea bpintea bpintea approved these changes

Assignees

No one assigned

Labels

:Analytics/SQL SQL querying >enhancement v7.7.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SQL: TIMESTAMP literal parsing failure when compared against DATETIME

6 participants

@matriv @elasticmachine @costin @astefan @bpintea @jakelandis