Skip to content

Commit 6d1815e

Browse files
zhengruifengdongjoon-hyun
authored andcommitted
[SPARK-49718][PS] Switch Scatter plot to sampled data
### What changes were proposed in this pull request? Switch `Scatter` plot to sampled data ### Why are the changes needed? when the data distribution has relationship with the order, the first n rows will not be representative of the whole dataset for example: ``` import pandas as pd import numpy as np import pyspark.pandas as ps # ps.set_option("plotting.max_rows", 10000) np.random.seed(123) pdf = pd.DataFrame(np.random.randn(10000, 4), columns=list('ABCD')).sort_values("A") psdf = ps.DataFrame(pdf) psdf.plot.scatter(x='B', y='A') ``` all 10k datapoints: ![image](https://github.com/user-attachments/assets/72cf7e97-ad10-41e0-a8a6-351747d5285f) before (first 1k datapoints): ![image](https://github.com/user-attachments/assets/1ed50d2c-7772-4579-a84c-6062542d9367) after (sampled 1k datapoints): ![image](https://github.com/user-attachments/assets/6c684cba-4119-4c38-8228-2bedcdeb9e59) ### Does this PR introduce _any_ user-facing change? yes ### How was this patch tested? ci and manually test ### Was this patch authored or co-authored using generative AI tooling? no Closes #48164 from zhengruifeng/ps_scatter_sampling. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 92cad2a commit 6d1815e

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

python/pyspark/pandas/plot/core.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -479,7 +479,7 @@ class PandasOnSparkPlotAccessor(PandasObject):
479479
"pie": TopNPlotBase().get_top_n,
480480
"bar": TopNPlotBase().get_top_n,
481481
"barh": TopNPlotBase().get_top_n,
482-
"scatter": TopNPlotBase().get_top_n,
482+
"scatter": SampledPlotBase().get_sampled,
483483
"area": SampledPlotBase().get_sampled,
484484
"line": SampledPlotBase().get_sampled,
485485
}

0 commit comments

Comments
 (0)