Commit 3a156b8
committed
SPARK-3211 .take() is OOM-prone with empty partitions
Instead of jumping straight from 1 partition to all partitions, do exponential
growth and double the number of partitions to attempt each time instead.
Fix proposed by Paul Nepywoda1 parent fb0db77 commit 3a156b8
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1068 | 1068 | | |
1069 | 1069 | | |
1070 | 1070 | | |
1071 | | - | |
| 1071 | + | |
1072 | 1072 | | |
1073 | 1073 | | |
1074 | 1074 | | |
| |||
0 commit comments