Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/sql-ref-ansi-compliance.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,6 @@ The behavior of some SQL functions can be different under ANSI mode (`spark.sql.
The behavior of some SQL operators can be different under ANSI mode (`spark.sql.ansi.enabled=true`).
- `array_col[index]`: This operator throws `ArrayIndexOutOfBoundsException` if using invalid indices.
- `map_col[key]`: This operator throws `NoSuchElementException` if key does not exist in map.
- `GROUP BY`: aliases in a select list can not be used in GROUP BY clauses. Each column referenced in a GROUP BY clause shall unambiguously reference a column of the table resulting from the FROM clause.

### Useful Functions for ANSI Mode

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1948,7 +1948,7 @@ class Analyzer(override val catalogManager: CatalogManager)
// mayResolveAttrByAggregateExprs requires the TreePattern UNRESOLVED_ATTRIBUTE.
_.containsAllPatterns(AGGREGATE, UNRESOLVED_ATTRIBUTE), ruleId) {
case agg @ Aggregate(groups, aggs, child)
if allowGroupByAlias && child.resolved && aggs.forall(_.resolved) &&
if conf.groupByAliases && child.resolved && aggs.forall(_.resolved) &&
groups.exists(!_.resolved) =>
agg.copy(groupingExpressions = mayResolveAttrByAggregateExprs(groups, aggs, child))
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -240,17 +240,6 @@ object SQLConf {
.intConf
.createWithDefault(100)

val ANSI_ENABLED = buildConf("spark.sql.ansi.enabled")
.doc("When true, Spark SQL uses an ANSI compliant dialect instead of being Hive compliant. " +
"For example, Spark will throw an exception at runtime instead of returning null results " +
"when the inputs to a SQL operator/function are invalid." +
"For full details of this dialect, you can find them in the section \"ANSI Compliance\" of " +
"Spark's documentation. Some ANSI dialect features may be not from the ANSI SQL " +
"standard directly, but their behaviors align with ANSI SQL's style")
.version("3.0.0")
.booleanConf
.createWithDefault(false)

val OPTIMIZER_EXCLUDED_RULES = buildConf("spark.sql.optimizer.excludedRules")
.doc("Configures a list of rules to be disabled in the optimizer, in which the rules are " +
"specified by their rule names and separated by comma. It is not guaranteed that all the " +
Expand Down Expand Up @@ -1221,9 +1210,8 @@ object SQLConf {
.createWithDefault(true)

val GROUP_BY_ALIASES = buildConf("spark.sql.groupByAliases")
.doc("This configuration is only effective when ANSI mode is disabled. When it is true and " +
s"${ANSI_ENABLED.key} is false, aliases in a select list can be used in group by clauses. " +
"Otherwise, an analysis exception is thrown in the case.")
.doc("When true, aliases in a select list can be used in group by clauses. When false, " +
"an analysis exception is thrown in the case.")
.version("2.2.0")
.booleanConf
.createWithDefault(true)
Expand Down Expand Up @@ -2547,6 +2535,17 @@ object SQLConf {
.checkValues(StoreAssignmentPolicy.values.map(_.toString))
.createWithDefault(StoreAssignmentPolicy.ANSI.toString)

val ANSI_ENABLED = buildConf("spark.sql.ansi.enabled")
.doc("When true, Spark SQL uses an ANSI compliant dialect instead of being Hive compliant. " +
"For example, Spark will throw an exception at runtime instead of returning null results " +
"when the inputs to a SQL operator/function are invalid." +
"For full details of this dialect, you can find them in the section \"ANSI Compliance\" of " +
"Spark's documentation. Some ANSI dialect features may be not from the ANSI SQL " +
"standard directly, but their behaviors align with ANSI SQL's style")
.version("3.0.0")
.booleanConf
.createWithDefault(false)

val SORT_BEFORE_REPARTITION =
buildConf("spark.sql.execution.sortBeforeRepartition")
.internal()
Expand Down

This file was deleted.

Loading