Skip to content

Commit 8229e15

Browse files
committed
[SPARK-23238][SQL] Externalize SQLConf configurations exposed in documentation
## What changes were proposed in this pull request? This PR proposes to expose few internal configurations found in the documentation. Also it fixes the description for `spark.sql.execution.arrow.enabled`. It's quite self-explanatory. ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes #20403 from HyukjinKwon/minor-doc-arrow. (cherry picked from commit 39d2c6b) Signed-off-by: hyukjinkwon <[email protected]>
1 parent 5dda5db commit 8229e15

File tree

1 file changed

+5
-7
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

1 file changed

+5
-7
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -123,14 +123,12 @@ object SQLConf {
123123
.createWithDefault(10)
124124

125125
val COMPRESS_CACHED = buildConf("spark.sql.inMemoryColumnarStorage.compressed")
126-
.internal()
127126
.doc("When set to true Spark SQL will automatically select a compression codec for each " +
128127
"column based on statistics of the data.")
129128
.booleanConf
130129
.createWithDefault(true)
131130

132131
val COLUMN_BATCH_SIZE = buildConf("spark.sql.inMemoryColumnarStorage.batchSize")
133-
.internal()
134132
.doc("Controls the size of batches for columnar caching. Larger batch sizes can improve " +
135133
"memory utilization and compression, but risk OOMs when caching data.")
136134
.intConf
@@ -1043,11 +1041,11 @@ object SQLConf {
10431041

10441042
val ARROW_EXECUTION_ENABLE =
10451043
buildConf("spark.sql.execution.arrow.enabled")
1046-
.internal()
1047-
.doc("Make use of Apache Arrow for columnar data transfers. Currently available " +
1048-
"for use with pyspark.sql.DataFrame.toPandas with the following data types: " +
1049-
"StringType, BinaryType, BooleanType, DoubleType, FloatType, ByteType, IntegerType, " +
1050-
"LongType, ShortType")
1044+
.doc("When true, make use of Apache Arrow for columnar data transfers. Currently available " +
1045+
"for use with pyspark.sql.DataFrame.toPandas, and " +
1046+
"pyspark.sql.SparkSession.createDataFrame when its input is a Pandas DataFrame. " +
1047+
"The following data types are unsupported: " +
1048+
"MapType, ArrayType of TimestampType, and nested StructType.")
10511049
.booleanConf
10521050
.createWithDefault(false)
10531051

0 commit comments

Comments
 (0)