Improved GraphX PageRank Test Coverage #1228
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Merged build triggered. |
|
Merged build started. |
Contributor
Author
|
@ankurdave thanks for pointing out this bug! |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16151/ |
Contributor
|
Jenkins, test this please |
Contributor
|
test this please |
|
QA tests have started for PR 1228 at commit
|
|
QA tests have finished for PR 1228 at commit
|
Contributor
|
@jegonzal seems like these GraphX tests actually fail -- any ideas what's going on? |
Contributor
Author
|
Yes. This is an extension of the unit tests to catch a class of bugs addressed in PR #1217 (which has not been merged). I believe @ankurdave was working on a merge of these two pull requests. |
|
Can one of the admins verify this patch? |
jegonzal
force-pushed
the
SimpleChainTest
branch
from
487c968 to
8498483
Compare
Contributor
Author
|
This should now be addressed in the latest master and does not depend on PR #1217 |
Contributor
Author
|
ok to test |
Contributor
|
Jenkins, test this please. |
|
Test build #23072 has started for PR 1228 at commit
|
|
Test build #23072 has finished for PR 1228 at commit
|
|
Test PASSed. |
Contributor
Author
|
@ankurdave and @rxin can we merge this now? |
|
Test build #24257 has started for PR 1228 at commit
|
|
Test build #24257 has finished for PR 1228 at commit
|
|
Test PASSed. |
Member
|
@jegonzal cleaning house a bit -- this is the oldest oustanding PR. Are you proceeding with it? |
Contributor
Author
|
I think we have covered most of this code in later tests (PR #1217) and the remaining tests need to be substantially updated which I can do in a later PR. I am going to go ahead and close this one. Sorry about the delay. |
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Dec 8, 2021
…e#1228) ### What changes were proposed in this pull request? This PR adds connector APIs for DELETE/UPDATE/MERGE operations. It is similar to what we had in 3.1 but with changes to support merge-on-read per [this](https://docs.google.com/document/d/12Ywmc47j3l2WF4anG5vL4qlrhT2OKigb7_EbIKhxg60/) design doc. ### Why are the changes needed? These changes are needed to support DELETE/UPDATE/MERGE operations. ### Does this PR introduce _any_ user-facing change? This PR adds isolated connector APIs that will be used only by Iceberg. ### How was this patch tested? Local testing.
turboFei
pushed a commit
to turboFei/spark
that referenced
this pull request
Nov 6, 2025
Remove Hive 0.12, 0.13, 0.14, 1.0, 1.1 support: <img width="1019" alt="image" src="https://media.github.corp.ebay.com/user/23251/files/2f3cc629-868c-4176-9aa6-40e3d339a05f"> ``` <p style="box-sizing: inherit; font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; line-height: var(--line-height-base); color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span style="box-sizing: inherit; font-weight: bold;">org.apache.spark.sql.hive.client.HiveClientSuite.0.14: renamePartitions</span><span> </span>(from org.apache.spark.sql.hive.client.HiveClientSuites)</p><div style="box-sizing: inherit; color: rgb(51, 51, 51); font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; text-align: right;">Failing for the past 1 build (Since<span> </span><a href="https://ci.altus.vip.ebay.com/carmel-8958/job/spark-sql-hive-3.5/1228/" class="model-link" style="box-sizing: inherit; text-decoration: var(--link-text-decoration); font-weight: var(--link-font-weight); color: var(--link-color);"><span class="build-status-icon__wrapper icon-red icon-sm" style="box-sizing: inherit; width: 16px; height: 16px; color: var(--danger); fill: var(--danger); opacity: 1; display: inline-flex; position: relative;"><span class="build-status-icon__outer" style="box-sizing: inherit; display: flex; position: absolute; top: 8px; left: 8px; width: 16px; height: 16px; transform: translate(-50%, -50%);"><svg viewBox="0 0 24 24" focusable="false" class="svg-icon "><use href="/carmel-8958/images/build-status/build-status-sprite.svg#build-status-static"></use></svg></span><svg viewBox="0 0 24 24" focusable="false" class="svg-icon icon-red icon-sm"><use href="/carmel-8958/static/c9b6471f/images/build-status/build-status-sprite.svg#last-failed"></use></svg></span><span class="jenkins-icon-adjacent" style="box-sizing: inherit; margin-left: 0.5rem; width: 100%;">apache#1228</span></a><span> </span>)</div><div style="box-sizing: inherit; color: rgb(51, 51, 51); font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; text-align: right;"><a href="https://ci.altus.vip.ebay.com/carmel-8958/job/spark-sql-hive-3.5/1228/testReport/junit/org.apache.spark.sql.hive.client/HiveClientSuite/0_14__renamePartitions/history" style="box-sizing: inherit; text-decoration: var(--link-text-decoration); font-weight: var(--link-font-weight); color: var(--link-color);">Took 0.16 sec.</a></div><div id="description" style="box-sizing: inherit; color: rgb(51, 51, 51); font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div style="box-sizing: inherit;"></div><div class="jenkins-buttons-row jenkins-buttons-row--invert" style="box-sizing: inherit; display: flex; align-items: center; justify-content: flex-end;"><a class="yui-button link-button" id="description-link" href="https://ci.altus.vip.ebay.com/carmel-8958/job/spark-sql-hive-3.5/1228/testReport/junit/org.apache.spark.sql.hive.client/HiveClientSuite/0_14__renamePartitions/editDescription" onclick="return replaceDescription();" style="box-sizing: border-box; text-decoration: none; font-weight: var(--btn-link-font-weight); display: inline-flex; color: var(--primary); margin-left: 0.25rem; margin-right: 0.25rem; padding: 0.375rem 1rem; min-width: 4.5rem; min-height: 2rem; cursor: pointer; user-select: none; font-size: var(--btn-font-size); line-height: var(--btn-line-height); text-align: center; vertical-align: middle; border: 2px solid transparent; border-radius: 4px; transition: color 0.15s ease-in-out 0s, background-color 0.15s ease-in-out 0s, border-color 0.15s ease-in-out 0s; background-color: transparent; align-items: center; justify-content: center;"><span class="leading-icon" style="box-sizing: inherit; display: inline-block; margin-right: 0.25rem;"><svg viewBox="0 0 16 16" focusable="false" class="svg-icon icon-small"><use href="/carmel-8958/static/c9b6471f/images/material-icons/edit.svg#edit"></use></svg></span>Add description</a></div></div> <h3 style="box-sizing: inherit; line-height: var(--line-height-heading); font-weight: bold; display: block; margin-top: 0.5rem; margin-bottom: 0.5rem; font-size: 1.5rem; color: rgb(51, 51, 51); font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Error Message</h3><pre style="box-sizing: inherit; white-space: pre-wrap; overflow-wrap: break-word; margin: 0px; font-size: var(--font-size-monospace); color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename partition. Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus</pre><h3 style="box-sizing: inherit; line-height: var(--line-height-heading); font-weight: bold; display: block; margin-top: 0.5rem; margin-bottom: 0.5rem; font-size: 1.5rem; color: rgb(51, 51, 51); font-family: -apple-system, blinkmacsystemfont, "Segoe UI", roboto, "Noto Sans", oxygen, ubuntu, cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Stacktrace</h3><pre style="box-sizing: inherit; white-space: pre-wrap; overflow-wrap: break-word; margin: 0px; font-size: var(--font-size-monospace); color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">sbt.ForkMain$ForkError: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename partition. Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus at org.apache.hadoop.hive.ql.metadata.Hive.renamePartition(Hive.java:607) at org.apache.spark.sql.hive.client.Shim_v0_12.renamePartition(HiveShim.scala:688) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$3(HiveClientImpl.scala:762) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$3$adapted(HiveClientImpl.scala:755) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$1(HiveClientImpl.scala:755) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$2(HiveClientImpl.scala:313) at org.apache.spark.sql.hive.client.HiveClientImpl.retry(HiveClientImpl.scala:247) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:293) at org.apache.spark.sql.hive.client.HiveClientImpl.updateCallMetrics(HiveClientImpl.scala:326) at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:322) at org.apache.spark.sql.hive.client.HiveClientImpl.renamePartitions(HiveClientImpl.scala:764) at org.apache.spark.sql.hive.client.HiveClientSuite.$anonfun$new$53(HiveClientSuite.scala:482) at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127) at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282) at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231) at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230) at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69) at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227) at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) at scala.collection.immutable.List.foreach(List.scala:431) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) at org.scalatest.Suite.run(Suite.scala:1114) at org.scalatest.Suite.run$(Suite.scala:1096) at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178) at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.scalatest.Suite.runNestedSuites(Suite.scala:1223) at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156) at org.scalatest.funsuite.AnyFunSuite.runNestedSuites(AnyFunSuite.scala:1564) at org.scalatest.Suite.run(Suite.scala:1111) at org.scalatest.Suite.run$(Suite.scala:1096) at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: sbt.ForkMain$ForkError: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1139) at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:221) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartition(HiveAlterHandler.java:398) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:3110) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:3083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy74.rename_partition(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.renamePartition(HiveMetaStoreClient.java:305) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90) at com.sun.proxy.$Proxy75.renamePartition(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.renamePartition(Hive.java:601) ... 83 more</pre>org.apache.spark.sql.hive.client.HiveClientSuite.0.14: renamePartitions (from org.apache.spark.sql.hive.client.HiveClientSuites) Failing for the past 1 build (Since [apache#1228](https://ci.altus.vip.ebay.com/carmel-8958/job/spark-sql-hive-3.5/1228/) ) [Took 0.16 sec.](https://ci.altus.vip.ebay.com/carmel-8958/job/spark-sql-hive-3.5/1228/testReport/junit/org.apache.spark.sql.hive.client/HiveClientSuite/0_14__renamePartitions/history) Error Message org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename partition. Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus Stacktrace sbt.ForkMain$ForkError: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename partition. Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus at org.apache.hadoop.hive.ql.metadata.Hive.renamePartition(Hive.java:607) at org.apache.spark.sql.hive.client.Shim_v0_12.renamePartition(HiveShim.scala:688) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$3(HiveClientImpl.scala:762) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$3$adapted(HiveClientImpl.scala:755) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$renamePartitions$1(HiveClientImpl.scala:755) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$2(HiveClientImpl.scala:313) at org.apache.spark.sql.hive.client.HiveClientImpl.retry(HiveClientImpl.scala:247) at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:293) at org.apache.spark.sql.hive.client.HiveClientImpl.updateCallMetrics(HiveClientImpl.scala:326) at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:322) at org.apache.spark.sql.hive.client.HiveClientImpl.renamePartitions(HiveClientImpl.scala:764) at org.apache.spark.sql.hive.client.HiveClientSuite.$anonfun$new$53(HiveClientSuite.scala:482) at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127) at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282) at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231) at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230) at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69) at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227) at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) at scala.collection.immutable.List.foreach(List.scala:431) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) at org.scalatest.Suite.run(Suite.scala:1114) at org.scalatest.Suite.run$(Suite.scala:1096) at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178) at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.scalatest.Suite.runNestedSuites(Suite.scala:1223) at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156) at org.scalatest.funsuite.AnyFunSuite.runNestedSuites(AnyFunSuite.scala:1564) at org.scalatest.Suite.run(Suite.scala:1111) at org.scalatest.Suite.run$(Suite.scala:1096) at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: sbt.ForkMain$ForkError: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: java.lang.UnsupportedOperationException RawLocalFileSystem doesn't support getAclStatus at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1139) at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:221) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartition(HiveAlterHandler.java:398) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:3110) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:3083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy74.rename_partition(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.renamePartition(HiveMetaStoreClient.java:305) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90) at com.sun.proxy.$Proxy75.renamePartition(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.renamePartition(Hive.java:601) ... 83 more ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds additional tests that identify a bug in the current implementation of
staticPagerank. This bug is addressed in #1217.