Skip to content

[SPARK-3058] [SQL] Support EXTENDED for EXPLAIN #1962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-3058] [SQL] Support EXTENDED for EXPLAIN #1962

wants to merge 2 commits into from

Conversation

@chenghao-intel
Copy link
Contributor

@chenghao-intel chenghao-intel commented Aug 15, 2014

Provide extended keyword support for explain command in SQL. e.g.

explain extended select key as a1, value as a2 from src where key=1;
== Parsed Logical Plan ==
Project ['key AS a1#3,'value AS a2#4]
 Filter ('key = 1)
  UnresolvedRelation None, src, None

== Analyzed Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = CAST(1, DoubleType))
  MetastoreRelation default, src, None

== Optimized Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  MetastoreRelation default, src, None

== Physical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  HiveTableScan [key#8,value#9], (MetastoreRelation default, src, None), None

Code Generation: false
== RDD ==
(2) MappedRDD[14] at map at HiveContext.scala:350
  MapPartitionsRDD[13] at mapPartitions at basicOperators.scala:42
  MapPartitionsRDD[12] at mapPartitions at basicOperators.scala:57
  MapPartitionsRDD[11] at mapPartitions at TableReader.scala:112
  MappedRDD[10] at map at TableReader.scala:240
  HadoopRDD[9] at HadoopRDD at TableReader.scala:230

It's the sub task of #1847. But can go without any dependency.

@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have started for PR 1962. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18594/consoleFull

@chenghao-intel chenghao-intel mentioned this pull request Aug 15, 2014
Closed
@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have started for PR 1962 at commit 295db74.

  • This patch merges cleanly.

@marmbrus
Copy link
Contributor

marmbrus commented Aug 16, 2014

test this please

@SparkQA
Copy link

SparkQA commented Aug 16, 2014

QA tests have started for PR 1962 at commit 295db74.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 16, 2014

QA tests have finished for PR 1962 at commit 295db74.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • abstract class Serializer
    • abstract class SerializerInstance
    • abstract class SerializationStream
    • abstract class DeserializationStream
    • class ShuffleBlockManager(blockManager: BlockManager,
    • case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command

@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented Aug 17, 2014

test this please

@SparkQA
Copy link

SparkQA commented Aug 17, 2014

QA tests have started for PR 1962 at commit 295db74.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 17, 2014

QA tests have finished for PR 1962 at commit 295db74.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command

@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented Aug 20, 2014

test this please.

@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented Aug 21, 2014

test this please

@SparkQA
Copy link

SparkQA commented Aug 21, 2014

QA tests have started for PR 1962 at commit 295db74.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 21, 2014

QA tests have finished for PR 1962 at commit 295db74.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • shift # Ignore main class (org.apache.spark.deploy.SparkSubmit) and use our own
    • case class SparkListenerTaskStart(stageId: Int, stageAttemptId: Int, taskInfo: TaskInfo)
    • case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command

@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented Aug 21, 2014

@marmbrus I think this is ready to be merged.

@marmbrus
Copy link
Contributor

marmbrus commented Aug 26, 2014

Thanks! I've merged this into master and 1.1

asfgit pushed a commit that referenced this pull request Aug 26, 2014
@chenghao-intel @marmbrus
[SPARK-3058] [SQL] Support EXTENDED for EXPLAIN
f8ac8ed 
Provide `extended` keyword support for `explain` command in SQL. e.g.
```
explain extended select key as a1, value as a2 from src where key=1;
== Parsed Logical Plan ==
Project ['key AS a1#3,'value AS a2#4]
 Filter ('key = 1)
  UnresolvedRelation None, src, None

== Analyzed Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = CAST(1, DoubleType))
  MetastoreRelation default, src, None

== Optimized Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  MetastoreRelation default, src, None

== Physical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  HiveTableScan [key#8,value#9], (MetastoreRelation default, src, None), None

Code Generation: false
== RDD ==
(2) MappedRDD[14] at map at HiveContext.scala:350
  MapPartitionsRDD[13] at mapPartitions at basicOperators.scala:42
  MapPartitionsRDD[12] at mapPartitions at basicOperators.scala:57
  MapPartitionsRDD[11] at mapPartitions at TableReader.scala:112
  MappedRDD[10] at map at TableReader.scala:240
  HadoopRDD[9] at HadoopRDD at TableReader.scala:230
```

It's the sub task of #1847. But can go without any dependency.

Author: Cheng Hao <[email protected]>

Closes #1962 from chenghao-intel/explain_extended and squashes the following commits:

295db74 [Cheng Hao] Fix bug in printing the simple execution plan
48bc989 [Cheng Hao] Support EXTENDED for EXPLAIN

(cherry picked from commit 156eb39)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in 156eb39 Aug 26, 2014
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@chenghao-intel @conviva-zz
[SPARK-3058] [SQL] Support EXTENDED for EXPLAIN
9188710 
Provide `extended` keyword support for `explain` command in SQL. e.g.
```
explain extended select key as a1, value as a2 from src where key=1;
== Parsed Logical Plan ==
Project ['key AS a1#3,'value AS a2#4]
 Filter ('key = 1)
  UnresolvedRelation None, src, None

== Analyzed Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = CAST(1, DoubleType))
  MetastoreRelation default, src, None

== Optimized Logical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  MetastoreRelation default, src, None

== Physical Plan ==
Project [key#8 AS a1#3,value#9 AS a2#4]
 Filter (CAST(key#8, DoubleType) = 1.0)
  HiveTableScan [key#8,value#9], (MetastoreRelation default, src, None), None

Code Generation: false
== RDD ==
(2) MappedRDD[14] at map at HiveContext.scala:350
  MapPartitionsRDD[13] at mapPartitions at basicOperators.scala:42
  MapPartitionsRDD[12] at mapPartitions at basicOperators.scala:57
  MapPartitionsRDD[11] at mapPartitions at TableReader.scala:112
  MappedRDD[10] at map at TableReader.scala:240
  HadoopRDD[9] at HadoopRDD at TableReader.scala:230
```

It's the sub task of apache#1847. But can go without any dependency.

Author: Cheng Hao <[email protected]>

Closes apache#1962 from chenghao-intel/explain_extended and squashes the following commits:

295db74 [Cheng Hao] Fix bug in printing the simple execution plan
48bc989 [Cheng Hao] Support EXTENDED for EXPLAIN
@chenghao-intel chenghao-intel deleted the explain_extended branch October 9, 2014 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@chenghao-intel @SparkQA @marmbrus