Skip to content

[SPARK-26224][SQL] Advice the user when creating many project on subsequent calls to withColumn #23285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

[SPARK-26224][SQL] Advice the user when creating many project on subsequent calls to withColumn #23285

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
da2c82e
[SPARK-26224][SQL] Avoid creating many project on subsequent calls to…
mgaido91 Dec 11, 2018
a40db10
fix UT failures
mgaido91 Dec 11, 2018
fa25e2e
Revert "fix UT failures"
mgaido91 Dec 17, 2018
5162b13
Revert "[SPARK-26224][SQL] Avoid creating many project on subsequent …
mgaido91 Dec 17, 2018
e37c2d6
warning for too many withColumn
mgaido91 Dec 17, 2018
802cb9e
add conf
mgaido91 Jan 14, 2019
5d1fc00
address comment
mgaido91 Mar 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2137,6 +2137,11 @@ class Dataset[T] private[sql](
* `column`'s expression must only refer to attributes supplied by this Dataset. It is an
* error to add a column that refers to some other Dataset.
*
* Please notice that this method introduces a `Project`. This means that using it in loops in
* order to add several columns can generate very big plans which can cause huge performance
* issues and even `StackOverflowException`s. A much better alternative use `select` with the
* list of columns to add.
*
* @group untypedrel
* @since 2.0.0
*/
Expand Down