Skip to content

Commit 6a194f1

Browse files
committed
[SPARK-33212][FOLLOW-UP][BUILD] Uses provided properties for Hadoop client dependencies in root pom
### What changes were proposed in this pull request? This PR is a followup of #30701. It uses properties of `hadoop-client-api.artifact`, `hadoop-client-runtime.artifact` and `hadoop-client-minicluster.artifact` explicitly to set the dependencies and versions. Otherwise, it is logically incorrect. For example, if you build with Hadoop 2, this dependency becomes `hadoop-client-api:2.7.4` internally, which does not exist in Hadoop 2 (https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-api). ### Why are the changes needed? - To fix the logical incorrectness. - It fixes a potential issue: this actually caused an issue when `generate-sources` plugin is used together with Hadoop 2 by default, which attempts to pull 2.7.4 of `hadoop-client-api`, `hadoop-client-runtime` and `hadoop-client-minicluster` for whatever reason. ### Does this PR introduce _any_ user-facing change? No for users and dev. It's more a cleanup. ### How was this patch tested? Manually checked the dependencies are correctly placed. Closes #31467 from HyukjinKwon/SPARK-33212. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 42c32e8 commit 6a194f1

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

pom.xml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1079,19 +1079,19 @@
10791079
<!-- Hadoop 3.x dependencies -->
10801080
<dependency>
10811081
<groupId>org.apache.hadoop</groupId>
1082-
<artifactId>hadoop-client-api</artifactId>
1082+
<artifactId>${hadoop-client-api.artifact}</artifactId>
10831083
<version>${hadoop.version}</version>
10841084
<scope>${hadoop.deps.scope}</scope>
10851085
</dependency>
10861086
<dependency>
10871087
<groupId>org.apache.hadoop</groupId>
1088-
<artifactId>hadoop-client-runtime</artifactId>
1088+
<artifactId>${hadoop-client-runtime.artifact}</artifactId>
10891089
<version>${hadoop.version}</version>
10901090
<scope>${hadoop.deps.scope}</scope>
10911091
</dependency>
10921092
<dependency>
10931093
<groupId>org.apache.hadoop</groupId>
1094-
<artifactId>hadoop-client-minicluster</artifactId>
1094+
<artifactId>${hadoop-client-minicluster.artifact}</artifactId>
10951095
<version>${yarn.version}</version>
10961096
<scope>test</scope>
10971097
</dependency>

0 commit comments

Comments
 (0)