Skip to content

[SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatibilitySuite failure due to test dependency on avro #16795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatibilitySuite failure due to test dependency on avro #16795

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2cc8653
[SPARK-19409][BUILD] Fix ParquetAvroCompatibilitySuite failure due to…
dongjoon-hyun Feb 3, 2017
499f6fd
Move to parent pom.
dongjoon-hyun Feb 6, 2017
42ff642
Revert "Move to parent pom."
dongjoon-hyun Feb 6, 2017
941bf00
Add spark-tags into mesos module.
dongjoon-hyun Feb 6, 2017
47e9033
Use 1.8.1 instead of 1.8.0.
dongjoon-hyun Feb 7, 2017
b9e336f
Add comments.
dongjoon-hyun Feb 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions resource-managers/mesos/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Feb 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed to fix the following error due to org.apache.spark.tags.ExtendedYarnTest.

[info] Running Spark tests using Maven with these arguments:  -Phadoop-2.3 -Phive -Pyarn -Pmesos -Phive-thriftserver -Pkinesis-asl -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest test --fail-at-end
...
[INFO] Spark Project Mesos ................................ FAILURE [ 10.687 s]
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project spark-mesos_2.11: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: There was an error in the forked process
[ERROR] java.lang.RuntimeException: Unable to load category: org.apache.spark.tags.ExtendedYarnTest
[ERROR] at org.apache.maven.surefire.group.match.SingleGroupMatcher.loadGroupClasses(SingleGroupMatcher.java:139)
[ERROR] at ...
[ERROR] Caused by: java.lang.ClassNotFoundException: org.apache.spark.tags.ExtendedYarnTest
...


<dependency>
<groupId>org.apache.mesos</groupId>
<artifactId>mesos</artifactId>
Expand Down
13 changes: 13 additions & 0 deletions sql/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,19 @@
<artifactId>parquet-avro</artifactId>
<scope>test</scope>
</dependency>
<!--
This version of avro test-dep is different from the one defined
in the parent pom. The parent pom has avro 1.7.7 test-dep for Hadoop.
Here, ParquetAvroCompatibilitySuite uses parquet-avro's AvroParquetWriter
which uses avro 1.8.0+ specific API. In Maven 3, we need to have
this here to have different versions for the same artifact.
-->
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Feb 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen . This is the added comments.

<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
Expand Down