[SPARK-20926][SQL] Removing exposures to guava library caused by directly accessing SessionCatalog's tableRelationCache #18148

rezasafi · 2017-05-30T22:22:32Z

There could be test failures because DataStorageStrategy, HiveMetastoreCatalog and also HiveSchemaInferenceSuite were exposed to guava library by directly accessing SessionCatalog's tableRelationCacheg. These failures occur when guava shading is in place.

What changes were proposed in this pull request?

This change removes those guava exposures by introducing new methods in SessionCatalog and also changing DataStorageStrategy, HiveMetastoreCatalog and HiveSchemaInferenceSuite so that they use those proxy methods.

How was this patch tested?

Unit tests passed after applying these changes.

…ly accessing SessionCatalog's tableRelationCache There were test failures because DataStorageStrategy, HiveMetastoreCatalog and also HiveSchemaInferenceSuite were exposed to the shaded Guava library. This change removes those exposures by introducing new methods in SessionCatalog.

vanzin · 2017-06-01T23:20:02Z

ok to test

vanzin

@marmbrus as far as I can tell, SessionCatalog is a public API right? If so we shouldn't ship 2.2 without fixing this, since it's exposing Guava.

vanzin · 2017-06-01T23:20:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+    * Accessing tableRelationCache directly is not recommended,
+    * since it will introduce exposures to guava libraries.
+    */
  val tableRelationCache: Cache[QualifiedTableName, LogicalPlan] = {


This seems to have been introduced in 2.2, can you make it private instead, and change all call sites?

SparkQA · 2017-06-01T23:24:15Z

Test build #77659 has finished for PR 18148 at commit 8253bbe.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-06-01T23:30:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

  }

+  /**
+    * This method provides a way to get a cached plan


The "without exposing components to Guava" part of all these comments is unnecessary.

zsxwing · 2017-06-01T23:45:14Z

@vanzin the whole catalyst project is private.

vanzin · 2017-06-01T23:47:54Z

Ah, yes. I read 'filter' instead of 'filterNot' in SparkBuild.scala where it sets mimaProjects.

SparkQA · 2017-06-02T01:44:34Z

Test build #77664 has finished for PR 18148 at commit 9821ea1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-02T06:13:28Z

Test build #77666 has finished for PR 18148 at commit 9421372.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin

A couple of suggestions for the names but otherwise LGTM.

vanzin · 2017-06-02T23:16:36Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+  }
+
+  /** This method provides a way to get a cached plan if the key exists. */
+  def getCachedTableIfPresent(key: QualifiedTableName): LogicalPlan = {


getCachedTable instead. I'd even just use cachedTable but this class at least seems to use verbs more than the rest of the code.

vanzin · 2017-06-02T23:16:45Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+  }
+
+  /** This method provides a way to cache a plan. */
+  def putTableInCache(t: QualifiedTableName, l: LogicalPlan): Unit = {


cacheTable

SparkQA · 2017-06-03T01:26:09Z

Test build #77696 has finished for PR 18148 at commit 2832253.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rezasafi · 2017-06-03T06:14:41Z

Not clear why the tests were failed.

vanzin · 2017-06-03T19:30:01Z

Fun, but probably unrelated:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f115a81f0e9, pid=125279, tid=139711419225856
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x88a0e9]  LoadKlassNode::make(PhaseGVN&, Node*, Node*, Node*, TypePtr const*, TypeKlassPtr const*)+0x49
#

vanzin · 2017-06-03T19:30:08Z

retest this please

SparkQA · 2017-06-03T21:59:25Z

Test build #77704 has finished for PR 18148 at commit 2832253.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-06-05T20:18:40Z

Merging to master.

…ctly accessing SessionCatalog's tableRelationCache There could be test failures because DataStorageStrategy, HiveMetastoreCatalog and also HiveSchemaInferenceSuite were exposed to guava library by directly accessing SessionCatalog's tableRelationCacheg. These failures occur when guava shading is in place. ## What changes were proposed in this pull request? This change removes those guava exposures by introducing new methods in SessionCatalog and also changing DataStorageStrategy, HiveMetastoreCatalog and HiveSchemaInferenceSuite so that they use those proxy methods. ## How was this patch tested? Unit tests passed after applying these changes. Author: Reza Safi <[email protected]> Closes #18148 from rezasafi/branch-2.2.

vanzin · 2017-06-06T16:51:43Z

@rezasafi please close this.

vanzin · 2017-06-06T16:53:04Z

Oh crap, I didn't notice the branch. @rezasafi in the future, always send PRs against the master branch first.

rezasafi · 2017-06-06T17:01:41Z

Sorry about this @vanzin. I didn't know that.

yhuai · 2017-06-08T04:36:03Z

@vanzin Seems merging to branch-2.2 was an accident? Since it is not really a bug fix, should we revert it from branch-2.2 and just keep it in the master?

vanzin · 2017-06-08T17:34:06Z

It was an accident but it shouldn't cause any harm either.

rezasafi changed the title ~~[SPARK-20926][SQL] Removing exposures to guava library caused by directly accessing SessionCatalog's tableRelationCacheg~~ [SPARK-20926][SQL] Removing exposures to guava library caused by directly accessing SessionCatalog's tableRelationCache May 30, 2017

vanzin reviewed Jun 1, 2017

View reviewed changes

Making tableRelationCache private and updating the comments.

9821ea1

Fixing scalastyle check errors

9421372

vanzin reviewed Jun 2, 2017

View reviewed changes

Changing the names for two of the methods.

2832253

rezasafi closed this Jun 6, 2017

rezasafi reopened this Jun 6, 2017

asfgit closed this in b61a401 Jun 6, 2017

[SPARK-20926][SQL] Removing exposures to guava library caused by directly accessing SessionCatalog's tableRelationCache #18148

[SPARK-20926][SQL] Removing exposures to guava library caused by directly accessing SessionCatalog's tableRelationCache #18148

Uh oh!

Conversation

rezasafi commented May 30, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

vanzin commented Jun 1, 2017

Uh oh!

vanzin left a comment

Choose a reason for hiding this comment

Uh oh!

vanzin Jun 1, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 1, 2017

Uh oh!

vanzin Jun 1, 2017

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Jun 1, 2017

Uh oh!

vanzin commented Jun 1, 2017

Uh oh!

SparkQA commented Jun 2, 2017

Uh oh!

SparkQA commented Jun 2, 2017

Uh oh!

vanzin left a comment

Choose a reason for hiding this comment

Uh oh!

vanzin Jun 2, 2017

Choose a reason for hiding this comment

Uh oh!

vanzin Jun 2, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 3, 2017

Uh oh!

rezasafi commented Jun 3, 2017

Uh oh!

vanzin commented Jun 3, 2017

Uh oh!

vanzin commented Jun 3, 2017

Uh oh!

SparkQA commented Jun 3, 2017

Uh oh!

vanzin commented Jun 5, 2017

Uh oh!

vanzin commented Jun 6, 2017

Uh oh!

vanzin commented Jun 6, 2017

Uh oh!

rezasafi commented Jun 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yhuai commented Jun 8, 2017

Uh oh!

vanzin commented Jun 8, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rezasafi commented Jun 6, 2017 •

edited

Loading