[SPARK-52394][PS] Fix autocorr divide-by-zero error under ANSI mode

### What changes were proposed in this pull request?
Fix autocorr divide-by-zero error under ANSI mode

### Why are the changes needed?
Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52169.

### Does this PR introduce _any_ user-facing change?
When ANSI is on,

FROM
```py
>>> s = ps.Series([1, 0, 0, 0])
>>> s.autocorr()
...
25/08/04 13:25:13 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 33)
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"corr" was called from
...
```

TO
```py
>>> s = ps.Series([1, 0, 0, 0])
>>> s.autocorr()
nan
```

### How was this patch tested?
Unit tests.

Commands below passed
```
 1004  SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.series.test_stat SeriesStatTests.test_autocorr"

 1009  SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.series.test_stat SeriesStatTests.test_autocorr
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #51192 from xinrong-meng/autocorr.

Authored-by: Xinrong Meng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>

Author: xinrong-meng

python/pyspark/pandas/series.py

@@ -3410,11 +3410,18 @@ def autocorr(self, lag: int = 1) -> float:
         else:
             lag_scol = F.lag(scol, lag).over(Window.orderBy(NATURAL_ORDER_COLUMN_NAME))
             lag_col_name = verify_temp_column_name(sdf, "__autocorr_lag_tmp_col__")
-            corr = (
-                sdf.withColumn(lag_col_name, lag_scol)
-                .select(F.corr(scol, F.col(lag_col_name)))
-                .head()[0]
-            )
+
+            sdf_lag = sdf.withColumn(lag_col_name, lag_scol)
+            if is_ansi_mode_enabled(sdf.sparkSession):
+                # Compute covariance between the original and lagged columns.
+                # If the covariance is None or zero (indicating no linear relationship),
+                # return NaN, otherwise, proceeding to compute correlation may raise
+                # DIVIDE_BY_ZERO under ANSI mode.
+                cov_value = sdf_lag.select(F.covar_samp(scol, F.col(lag_col_name))).head()[0]
+                if cov_value is None or cov_value == 0.0:
+                    return np.nan
+            corr = sdf_lag.select(F.corr(scol, F.col(lag_col_name))).head()[0]
+
         return np.nan if corr is None else corr
 
     @with_ansi_mode_context

python/pyspark/pandas/tests/series/test_stat.py

@@ -606,6 +606,9 @@ def test_autocorr(self):
         with self.assertRaisesRegex(TypeError, r"lag should be an int; however, got"):
             psser.autocorr(1.0)
 
+        psser = ps.Series([1, 0, 0, 0])
+        self.assertTrue(bool(np.isnan(psser.autocorr())))
+
     def _test_autocorr(self, pdf):
         psdf = ps.from_pandas(pdf)
         for lag in range(-10, 10):