Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented Apr 24, 2018

What changes were proposed in this pull request?

SPARK-22543 fixes the 64kb compile error for deeply nested expression for non-wholestage codegen. This PR extends it to support wholestage codegen.

This patch extracts necessary parameters for a deeply nested expression when it is split into a function.

TODO: In the future, this should be extended to splitExpressions too to automatically extract the current inputs and put them into the parameter list.

WIP: This is in WIP status for now. It brings up previous changes in #19813 with latest codebase. This will implement the proposal #19813 (comment) to overcome the limit in the previous PR.

How was this patch tested?

Added tests and existing tests.

@SparkQA
Copy link

SparkQA commented Apr 24, 2018

Test build #89778 has finished for PR 21140 at commit e413043.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 24, 2018

Test build #89786 has finished for PR 21140 at commit ae25004.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 25, 2018

Test build #89809 has finished for PR 21140 at commit 7dc6ccc.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 25, 2018

Test build #89816 has finished for PR 21140 at commit 85568e7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented Apr 25, 2018

The remaining test failure at org.apache.spark.sql.TPCDSQuerySuite.q97 is caused by using a statement as function parameter. This should be addressed by the proposal at #19813 (comment).

@viirya
Copy link
Member Author

viirya commented Apr 30, 2018

I created another PR for the proposal implementation at #21193. So it can be reviewed easily.

@viirya
Copy link
Member Author

viirya commented Jul 16, 2018

This feature can be largely simplified once codegen framework is revamped later. So I close it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants