Skip to content

Commit ced8730

Browse files
committed
[SPARK-53077][CORE][TESTS] Add an env to control the execution of the SparkBloomFilterSuite
### What changes were proposed in this pull request? This pr adds an environment variable named `SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED` to control whether the test case `SparkBloomFilterSuite` is executed. It also ensures that this test case is only run for validation in the daily tests specified in `build_non_ansi.yml`. ### Why are the changes needed? The `SparkBloomFilterSuite` requires periodic validation, but due to its excessively long execution time (over 10 minutes), it is not suitable for execution in the Change Pipeline. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual verification: - maven ``` build/mvn package --pl common/sketch [INFO] Running org.apache.spark.util.sketch.SparkBloomFilterSuite [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.001 s -- in org.apache.spark.util.sketch.SparkBloomFilterSuite ``` ``` SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED=true build/mvn package --pl common/sketch [INFO] Running org.apache.spark.util.sketch.SparkBloomFilterSuite [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 401.9 s -- in org.apache.spark.util.sketch.SparkBloomFilterSuite ``` - sbt ``` build/sbt clean "sketch/test" [info] Test run started (JUnit Jupiter) [info] Test org.apache.spark.util.sketch.SparkBloomFilterSuite ignored: Environment variable [SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED] does not exist [info] Test run finished: 0 failed, 0 ignored, 0 total, 0.016s ``` ``` SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED=true build/sbt clean "sketch/test" [info] Test run started (JUnit Jupiter) [info] Test org.apache.spark.util.sketch.SparkBloomFilterSuite#testAccuracyRandomDistribution(long, double, int, org.junit.jupiter.api.TestInfo):#1 started [info] Test org.apache.spark.util.sketch.SparkBloomFilterSuite#testAccuracyEvenOdd(long, double, int, org.junit.jupiter.api.TestInfo):#1 started [info] Test run finished: 0 failed, 0 ignored, 2 total, 359.776s ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51806 from LuciferYang/SPARK-53077. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
1 parent 6ef9a9d commit ced8730

File tree

2 files changed

+4
-1
lines changed

2 files changed

+4
-1
lines changed

.github/workflows/build_non_ansi.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ jobs:
4040
"PYSPARK_IMAGE_TO_TEST": "python-311",
4141
"PYTHON_TO_TEST": "python3.11",
4242
"SPARK_ANSI_SQL_MODE": "false",
43+
"SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED": "true"
4344
}
4445
jobs: >-
4546
{

common/sketch/src/test/java/org/apache/spark/util/sketch/SparkBloomFilterSuite.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.util.sketch;
1919

2020
import org.junit.jupiter.api.*;
21+
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
2122
import org.junit.jupiter.params.ParameterizedTest;
2223
import org.junit.jupiter.params.provider.Arguments;
2324
import org.junit.jupiter.params.provider.MethodSource;
@@ -33,7 +34,8 @@
3334
import java.util.stream.LongStream;
3435
import java.util.stream.Stream;
3536

36-
@Disabled("TODO(SPARK-53077): Re-enable with a resonable test time.")
37+
@EnabledIfEnvironmentVariable(
38+
named = "SPARK_TEST_SPARK_BLOOM_FILTER_SUITE_ENABLED", matches = "true")
3739
public class SparkBloomFilterSuite {
3840

3941
// the implemented fpp limit is only approximating the hard boundary,

0 commit comments

Comments
 (0)