Skip to content

Commit cc49244

Browse files
yaooqinnHyukjinKwon
authored andcommitted
[SPARK-31258][BUILD] Pin the avro version in SBT
### What changes were proposed in this pull request? add arvo dep in SparkBuild ### Why are the changes needed? fix sbt unidoc like #28017 (comment) ```scala [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/scala-2.12/unidoc... [info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/javaunidoc... [error] /home/jenkins/workspace/SparkPullRequestBuilder6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ^ [info] No documentation generated with unsuccessful compiler run [error] one error found ``` ### Does this PR introduce any user-facing change? no ### How was this patch tested? pass jenkins and verify manually with `sbt dependencyTree` ```scala kentyaohulk  ~/spark   dep  build/sbt dependencyTree | grep avro | grep -v Resolving [info] +-org.apache.avro:avro-mapred:1.8.2 [info] | +-org.apache.avro:avro-ipc:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] +-org.apache.avro:avro:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] org.apache.spark:spark-avro_2.12:3.1.0-SNAPSHOT [S] [info] | | | +-org.apache.avro:avro-mapred:1.8.2 [info] | | | | +-org.apache.avro:avro-ipc:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 [info] | | | +-org.apache.avro:avro:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 ``` Closes #28020 from yaooqinn/dep. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 336621e) Signed-off-by: HyukjinKwon <[email protected]>
1 parent 49dd32d commit cc49244

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

project/SparkBuild.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -624,7 +624,8 @@ object DependencyOverrides {
624624
dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
625625
dependencyOverrides += "commons-io" % "commons-io" % "2.4",
626626
dependencyOverrides += "xerces" % "xercesImpl" % "2.12.0",
627-
dependencyOverrides += "jline" % "jline" % "2.14.6")
627+
dependencyOverrides += "jline" % "jline" % "2.14.6",
628+
dependencyOverrides += "org.apache.avro" % "avro" % "1.8.2")
628629
}
629630

630631
/**

0 commit comments

Comments
 (0)