Skip to content

Conversation

@AngersZhuuuu
Copy link
Contributor

What changes were proposed in this pull request?

As mentioned in #29428 (comment) by @viirya ,
fix bug in UT, since in script transformation no-serde mode, output of decimal is same in both hive-1.2/hive-2.3

Why are the changes needed?

FIX UT

Does this PR introduce any user-facing change?

NO

How was this patch tested?

EXISTED UT

@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Aug 23, 2020

CC @viirya @maropu

@viirya viirya changed the title [SPARK-32608][SQL][3.0][FOLLOW-UP] Script Transform ROW FORMAT DELIMIT value should format value [SPARK-32608][SQL][3.0][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Script Transform ROW FORMAT DELIMIT value should format value Aug 23, 2020
@viirya
Copy link
Member

viirya commented Aug 23, 2020

retest this please.

@maropu
Copy link
Member

maropu commented Aug 23, 2020

since in script transformation no-serde mode, output of decimal is same in both hive-1.2/hive-2.3

Oh, I see. The fix LGTM

@SparkQA
Copy link

SparkQA commented Aug 23, 2020

Test build #127801 has finished for PR 29521 at commit 0fed503.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 23, 2020

Test build #127802 has finished for PR 29521 at commit 0fed503.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member

viirya commented Aug 23, 2020

retest this please

@viirya
Copy link
Member

viirya commented Aug 23, 2020

Looks good pending test.

@maropu
Copy link
Member

maropu commented Aug 23, 2020

@AngersZhuuuu btw, I think you'd better to run tests w/ [test-hadoop2.7][test-hive1.2] for your open TRANSFORM-related PRs, too.

@AngersZhuuuu
Copy link
Contributor Author

@AngersZhuuuu btw, I think you'd better to run tests w/ [test-hadoop2.7][test-hive1.2] for your open TRANSFORM-related PRs, too.

Yea, missing this point before.

@SparkQA
Copy link

SparkQA commented Aug 23, 2020

Test build #127803 has finished for PR 29521 at commit 0fed503.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AngersZhuuuu
Copy link
Contributor Author

@viirya the failed UT not realetd to this pr and seems existed issue, is there any jira refer to this problem or need to fix this .

@maropu
Copy link
Member

maropu commented Aug 23, 2020

I checked the commit history and #29409 and #29417 seem to be related to the failure: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127803/
@MaxGekk Could you check it? cc: @HyukjinKwon

@viirya
Copy link
Member

viirya commented Aug 23, 2020

@viirya the failed UT not realetd to this pr and seems existed issue, is there any jira refer to this problem or need to fix this .

Yes, they are known failure in hive-1.2. Please see https://issues.apache.org/jira/browse/SPARK-32689.

@viirya
Copy link
Member

viirya commented Aug 23, 2020

I checked the commit history and #29409 and #29417 seem to be related to the failure: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127803/
@MaxGekk Could you check it? cc: @HyukjinKwon

Yeah, seems so. @MaxGekk the following two tests are failed in branch-3.0 and master under hive-1.2 profile now.

org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite.Read/Write Hive PARQUET serde table
org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite.Read/Write Hive TEXTFILE serde table

Could you help check and fix them?

@viirya
Copy link
Member

viirya commented Aug 23, 2020

Jenkins shows this fixed ScriptTransformationSuite and others are known failures. Github Actions was passed except for CrossValidatorTests in PySpark which is caused by #29445. I'm going to merge this test fix. cc @dongjoon-hyun @HyukjinKwon @maropu

viirya pushed a commit that referenced this pull request Aug 23, 2020
…pt Transform ROW FORMAT DELIMIT value should format value

### What changes were proposed in this pull request?
As mentioned in #29428 (comment) by viirya ,
fix bug in UT, since in script transformation no-serde mode, output of decimal is same in both hive-1.2/hive-2.3

### Why are the changes needed?
FIX UT

### Does this PR introduce _any_ user-facing change?
NO

### How was this patch tested?
EXISTED UT

Closes #29521 from AngersZhuuuu/SPARK-32608-3.0-FOLLOW-UP.

Authored-by: angerszhu <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
@viirya
Copy link
Member

viirya commented Aug 23, 2020

Merging to branch-3.0. Thanks @AngersZhuuuu

@viirya viirya closed this Aug 23, 2020
@MaxGekk
Copy link
Member

MaxGekk commented Aug 23, 2020

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127803/
@MaxGekk Could you check it?

Here is the fix #29523

@dongjoon-hyun
Copy link
Member

Thank you, @AngersZhuuuu and all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants