Sort Processor does not have proper behavior with targetField #25237
Conversation
talevy
added
:Data Management/Ingest Node
|
Can you just always copy the list? Modifying a list from somewhere else is sneaky. Also, should be no need for the size check. |
|
@rjernst guess I was just preserving that "optimizing" logic, but yeah, super minimal. I'll simplify |
|
updated. thanks for looking @rjernst |
There was a problem hiding this comment.
We really should not have random logic like this anywhere in ES. Do we really need random field names to test whether sorting works? I would rather see explicit fields names "field1" and "field2" and actually test both sort orders than always have these random field names. They have their place, but for things where the field name matters.
|
There was another occurrence of this failure on 5.x gradle :modules:ingest-common:test -Dtests.seed=5FE7727E3E91839F -Dtests.class=org.elasticsearch.ingest.common.SortProcessorTests -Dtests.method="testSortWithTargetField" -Dtests.security.manager=true -Dtests.locale=fi-FI -Dtests.timezone=Asia/Omsk And master: gradle :modules:ingest-common:test -Dtests.seed=14E3C30AB5E521B6 -Dtests.class=org.elasticsearch.ingest.common.SortProcessorTests -Dtests.method="testSortWithTargetField" -Dtests.security.manager=true -Dtests.locale=tr-TR -Dtests.timezone=Asia/Damascus Both reproduced for me |
martijnvg
left a comment
martijnvg
left a comment
There was a problem hiding this comment.
LGTM, +1 to Ryan's test suggestion.
to specify a `targetField`. This results in some interesting behavior that was missed in the review. This processor sorts in-place, so there is a side-effect in both the original field and the target field. Another bug was that the targetField was not being set if the list being sorted was fewer than two elements. The new behavior works like this: If targetField and fieldName are not the same, we copy the list.
…25237) to specify a `targetField`. This results in some interesting behavior that was missed in the review. This processor sorts in-place, so there is a side-effect in both the original field and the target field. Another bug was that the targetField was not being set if the list being sorted was fewer than two elements. The new behavior works like this: If targetField and fieldName are not the same, we copy the list.
* master: [Test] restore BWC for parent-join now that the new mapping format is in 5.x Add a section named "relations" in the ParentJoinFieldMapper (elastic#25248) test: Ported more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests. (elastic#25173) fix: Sort Processor does not have proper behavior with targetField (elastic#25237) Allow reader wrappers to have different live docs but the same cache key.
…y-context * 'master' of github.com:elastic/elasticsearch: (21 commits) [DOCS] Clarify expected availability of HDFS for the HDFS Repository (elastic#25220) Remove some redundant 140 character checkstyle suppressions [Docs] more fix for the parent-join docs [Docs] Fix cross reference for parent-join field More advices around search speed and disk usage. (elastic#25252) Add documentation for the new parent-join field (elastic#25227) [analysis-icu] Allow setting unicodeSetFilter (elastic#20814) Introduce translog size and age based retention policies (elastic#25147) Add needs methods for specific variables to Painless script context factories. (elastic#25267) Improves snapshot logging and snapshoth deletion error handling (elastic#25264) Add unit test for PathHierarchyTokenizerFactory (elastic#24984) Deprecate tribe service Moved more token filters to analysis-common module. [Test] Make sure that SearchAfterSortedDocQueryTests uses a single threaded searcher [DOCS] Defined es-test-dir and plugins-examples-dir in index.asciidoc. (elastic#25232) Test fix - removed superfluous assertion (elastic#25247) [Test] restore BWC for parent-join now that the new mapping format is in 5.x Add a section named "relations" in the ParentJoinFieldMapper (elastic#25248) test: Ported more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests. (elastic#25173) fix: Sort Processor does not have proper behavior with targetField (elastic#25237) ...
clintongormley
changed the title
5 participants
#24133 added the ability to specify a
targetFieldin SortProcessor. This results in some interesting behavior that was missed in the review.This processor sorts in-place, so there is a side-effect in both the original field and the target field.
Another bug was that the targetField was not being set if the list being sorted was fewer than two elements.
The new behavior works like this: If targetField and fieldName are not the same, we copy the list.
to reproduce original test failure: