Query refactoring: TermQueryBuilder refactoring and test

[cbuescher]

Split the parse(QueryParseContext ctx) method into a parsing and a query building part, adding Streamable for serialization and hashCode(), equals() for better testing.
Add basic unit test for Builder and Parser.

PR goes agains query-refacoring feature branch.

[javanna]

why package private? Other members are private?

[cbuescher]

changed that in new version of this PR

[cbuescher]

Rebased this PR on top of current tip of feature branch and changed test to make use of new BaseQueryTestCase.

[javanna]

about these checks... we check the same right after parsing (at some point will be moved to coord node), is that correct? and we check again here (which will stay on the data node). I wonder if we should really repeat these checks, or share them between the two methods and introduce some validation method.

[cbuescher]

I added them here because in theory one could create query objects without parsing them. Maybe a void validate() on each query builder that also the parsers can use to check before they return a new builder from the fromXContent() method?

[cbuescher]

I added a validation method for TermQueryBuilder and reused that in the parser. Only downside of this is where previously we had a QueryParseException we now have a ElasticsearchNullPointerException, or we need to try/catch and rethrow which IMHO defeats the purpose. Have a look and let me know what you think.

[javanna]

First thing which I keep forgetting: you are right, queries can be created without going through fromXContent, java api will do just that. There are different places where we need the same validation, I think having a validate method is a good choice.

That said, one question is where do we call it from? fromXContent after parsing (on coordinating node in the future) makes perfect sense, but that is called only when the query is provided as json. Java api allows to create the intermediate query representation directly, meaning that the fromXContent step will be skipped there. What can happen in this case is 1) that the query gets serialized to other nodes without being validated, and validated on the data nodes multiple times leading to multiple errors, which is not quite what we want (we want to catch errors once and earlier), 2) that the coordinating node is the data node where the query gets executed, query might not even get serialized, we execute toQuery straight-away.

I believe the proper way to validate the query is hooking into the existing request validation mechanism (ActionRequest#validate). Every request gets already validated on the coordinating node. The SearchRequest should simply call query#validate then I think (only problem being that we cannot quite do it yet, we need to refactor the search request first as right now the whole search request is just a big json object)?

As for exceptions, if we adopt this approach I think we should be consistent with the existing validation infra and have ActionRequestValidationExceptions.

Does this make sense to you guys?

If so, we need to decide what to do for now, I think having the validate method on the base class makes sense as Lee mentioned. For now we could call it from a single place though (toQuery is safer as it covers all the cases), once we move to parsing to the coord node we need to move it to the validation api instead.

[dakrone]

+1 to this, I think this is a great ide, for now we can do the validate method and figure out where to hook it in at a later time.

Also as an aside, I'd love to get rid of Preconditions.checkNotNull and just use Objects.requireNonNull since it's part of the JVM, but that's just personal preference.

[cbuescher]

When I add this to the QueryBuilder interface, I'll have add an empty impl to the BaseQueryBuilder for now (can be removed later once all queries have validate() method). For the details I would like to open a separate issue in which we can then have discussion about exact signature, which exceptions to be thrown, Preconditions vs. Objects.requireNonNull etc... Let me know if you agree with this plan.

[cbuescher]

Opened #10777 for separate discussion on how to handle validation.

[javanna]

left a few comments

[cbuescher]

I pulled out the validation of the two important fields into a method shared by builder and parser part, but not sure if this is the best way to go. Happy about comments there. Hope I adressed the rest of your comments, I would prefer tracking the use of QueryParseContext in toQuery() in a separate issue before making huge changes there. Same goes for generifying named queries (lookup but also maybe simplification)

[dakrone]

Can you change name to fieldName in the constructor too?

[cbuescher]

done

[cbuescher]

Changed variable name in constructor and added validate() to the QueryBuilder interface. As long as this is not implemented by all queries, added empty impl to BaseQueryBuilder. Also opened two separate issues to keep track of further ideas for validation (#10777) and for generifying named queries (#10776).
I'm still not sure about how to improve the tests for toQuery() because the resulting lucene queries hide a lot of details, so assertions depend on explicit casts and knowing implementation details that might break often. I'm inclined to leave the toQuery()tests here very general and rely on other integration tests with real cluster setup / mappings etc. to check that queries work. Thoughts on this appreciated.

[dakrone]

Maybe add the @Override annotation here?

[cbuescher]

One question I had myself here: I had to change this for the fromXContentent() test to work. Otherwise, if the value is e.g. an int, it is written to the XContent query as a number value. With the original parser.text() it is then read in as String (which doesn't seem to matter for the later toQuery(), since there BytesRefs.toBytesRef(value) is used, which calls toString() anyway) but blows up the newly introduced equality test. I changes this to objectText() because this does some checking based in the JsonToken type.

[cbuescher]

Thinking about it, I tend to want to revert this change since I'm not sure about it's implications. The problem then is that Object value can be of different type in original query and after fromXContent(), but in the end it only matters that toQuery() produces the same result, and there every Object is converted to lucene BytesRef. Thats why I tend to use that conversion also in the TermQueryBuilder#equals(). Will push that change to show what I mean.

[javanna]

we discussed this and we went for always keeping BytesRef within TermQueryBuilder rather than mixing up String and BytesRef depending on whether we received the query via java api or we parsed it through json. Java api will still have a setter that accepts a string, but internally that setter will convert from String to BytesRef.

We do have to fix the above parser.text() which looks like a bug that never manifested as when parsing and executing on the same (data) node everything still works. But we are currenly parsing the value in the term query short format (term: { field: value}) always as a string.

[cbuescher]

@javanna went through the comments and current diff again, here are the things that are open from my point of view:

• using context.setMapUnmappedFieldAsString(true) in the BaseQueryTestCase
• using BytesRefs.toBytesRef(value) in equals() and hashCode() from my last update commit

Anything else open here that I'm missing?

[javanna]

I think we have to check if these are empty too and barf in that case?

[javanna]

one more thing: s/Fieldname/ field name
I think in general the first line of an error should be lowercase, that's pretty much the convention we use in the existing validation code.

[javanna]

if you rebase you'll get this change that was made upstream and merged back into our branch

[cbuescher]

Rebased this PR on current feature branch and changed validation exception class to own version similar to the previously suggested ActionRequestValidationException. Also opened separate issue #10974 for keeping track of adding tests for invalid json and set the TermQueryBuilder test to 20 repetitions.

[javanna]

need to update exception in javadocs

[javanna]

LGTM besides the minor comments left, if you can address them that would be great, this is good to merge then!

[cbuescher]

Thanks for the quick response, pushed this to the feature branch.

[cbuescher]

Thanks for the review and for closing the issue, somehow it got opened
again yesterday automatically.

On Wed, May 6, 2015 at 7:34 AM, Luca Cavanna [email protected]
wrote:

Closed #10669 #10669.

—
Reply to this email directly or view it on GitHub
#10669 (comment).

Christoph Büscher