Skip to content

Commit 51b9c2a

Browse files
committed
[SPARK-3039] [BUILD] Spark assembly for new hadoop API (hadoop 2) contains avro-mapred for
hadoop 1 API had been marked as resolved but did not work for at least some builds due to version conflicts using avro-mapred-1.7.5.jar and avro-mapred-1.7.6-hadoop2.jar (the correct version) when building for hadoop2. sql/hive/pom.xml org.spark-project.hive:hive-exec's depends on 1.7.5: Building Spark Project Hive 1.2.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-dependency-plugin:2.4:tree (default-cli) @ spark-hive_2.10 --- [INFO] org.apache.spark:spark-hive_2.10:jar:1.2.0 [INFO] +- org.spark-project.hive:hive-exec:jar:0.13.1a:compile [INFO] | \- org.apache.avro:avro-mapred:jar:1.7.5:compile [INFO] \- org.apache.avro:avro-mapred:jar:hadoop2:1.7.6:compile [INFO] Excluding this dependency allows the explicitly listed avro-mapred dependency to be picked up.
1 parent 6f34131 commit 51b9c2a

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

pom.xml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -938,6 +938,10 @@
938938
<groupId>com.esotericsoftware.kryo</groupId>
939939
<artifactId>kryo</artifactId>
940940
</exclusion>
941+
<exclusion>
942+
<groupId>org.apache.avro</groupId>
943+
<artifactId>avro-mapred</artifactId>
944+
</exclusion>
941945
</exclusions>
942946
</dependency>
943947
<dependency>

0 commit comments

Comments
 (0)