-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-22600][SQL][WIP] Fix 64kb limit for deeply nested expressions under wholestage codegen #21140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
… into parameter list.
|
Test build #89778 has finished for PR 21140 at commit
|
|
Test build #89786 has finished for PR 21140 at commit
|
|
Test build #89809 has finished for PR 21140 at commit
|
|
Test build #89816 has finished for PR 21140 at commit
|
|
The remaining test failure at |
|
I created another PR for the proposal implementation at #21193. So it can be reviewed easily. |
|
This feature can be largely simplified once codegen framework is revamped later. So I close it now. |
What changes were proposed in this pull request?
SPARK-22543 fixes the 64kb compile error for deeply nested expression for non-wholestage codegen. This PR extends it to support wholestage codegen.
This patch extracts necessary parameters for a deeply nested expression when it is split into a function.
TODO: In the future, this should be extended to
splitExpressionstoo to automatically extract the current inputs and put them into the parameter list.WIP: This is in WIP status for now. It brings up previous changes in #19813 with latest codebase. This will implement the proposal #19813 (comment) to overcome the limit in the previous PR.
How was this patch tested?
Added tests and existing tests.