Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6e70301
[SPARK-31636][SQL][DOCS] Remove HTML syntax in SQL reference
huaxingao May 4, 2020
fad1bd0
fix
huaxingao May 4, 2020
d3033f1
fix
huaxingao May 4, 2020
ca9f9eb
[MINOR][SQL][TESTS] Disable UI in SQL benchmarks by default
MaxGekk May 2, 2020
0c8146a
[SPARK-31571][R] Overhaul stop/message/warning calls to be more canon…
May 3, 2020
d8a2fa0
[SPARK-31527][SQL][TESTS][FOLLOWUP] Fix the number of rows in `DateTi…
MaxGekk May 4, 2020
da32137
[SPARK-31267][SQL] Flaky test: WholeStageCodegenSparkSubmitSuite.Gene…
tianshizz May 4, 2020
065871c
[SPARK-31626][SQL] Port HIVE-10415: hive.start.cleanup.scratchdir con…
wangyum May 4, 2020
931c0bc
[SPARK-31606][SQL] Reduce the perf regression of vectorized parquet r…
cloud-fan May 4, 2020
ffd69c6
[MINOR][DOCS] Fix typo in documents
kiszk May 4, 2020
78758b6
[SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the …
brkyvz May 4, 2020
45f939f
[SPARK-31633][BUILD] Upgrade SLF4J from 1.7.16 to 1.7.30
dongjoon-hyun May 4, 2020
6e340c8
Revert "[SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that lever…
huaxingao May 4, 2020
368cdc8
Revert "[MINOR][DOCS] Fix typo in documents"
huaxingao May 4, 2020
0bd2ca5
Revert "[SPARK-31606][SQL] Reduce the perf regression of vectorized p…
huaxingao May 4, 2020
d3d175b
Revert "[SPARK-31626][SQL] Port HIVE-10415: hive.start.cleanup.scratc…
huaxingao May 4, 2020
24eb041
Revert "[SPARK-31267][SQL] Flaky test: WholeStageCodegenSparkSubmitSu…
huaxingao May 4, 2020
8f9e9b2
Revert "[SPARK-31527][SQL][TESTS][FOLLOWUP] Fix the number of rows in…
huaxingao May 4, 2020
760dece
Revert "[SPARK-31571][R] Overhaul stop/message/warning calls to be mo…
huaxingao May 4, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions dev/deps/spark-deps-hadoop-2.7-hive-1.2
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
javolution/5.5.1//javolution-5.5.1.jar
jaxb-api/2.2.2//jaxb-api-2.2.2.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
jcl-over-slf4j/1.7.30//jcl-over-slf4j-1.7.30.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
jersey-client/2.30//jersey-client-2.30.jar
jersey-common/2.30//jersey-common-2.30.jar
Expand All @@ -135,7 +135,7 @@ json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
jsp-api/2.1//jsp-api-2.1.jar
jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.7.1//kubernetes-client-4.7.1.jar
kubernetes-model-common/4.7.1//kubernetes-model-common-4.7.1.jar
Expand Down Expand Up @@ -184,8 +184,8 @@ scala-reflect/2.12.10//scala-reflect-2.12.10.jar
scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
slf4j-api/1.7.30//slf4j-api-1.7.30.jar
slf4j-log4j12/1.7.30//slf4j-log4j12-1.7.30.jar
snakeyaml/1.24//snakeyaml-1.24.jar
snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
snappy/0.2//snappy-0.2.jar
Expand Down
8 changes: 4 additions & 4 deletions dev/deps/spark-deps-hadoop-2.7-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
javolution/5.5.1//javolution-5.5.1.jar
jaxb-api/2.2.2//jaxb-api-2.2.2.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
jcl-over-slf4j/1.7.30//jcl-over-slf4j-1.7.30.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
jersey-client/2.30//jersey-client-2.30.jar
jersey-common/2.30//jersey-common-2.30.jar
Expand All @@ -150,7 +150,7 @@ json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
jsp-api/2.1//jsp-api-2.1.jar
jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.7.1//kubernetes-client-4.7.1.jar
kubernetes-model-common/4.7.1//kubernetes-model-common-4.7.1.jar
Expand Down Expand Up @@ -198,8 +198,8 @@ scala-reflect/2.12.10//scala-reflect-2.12.10.jar
scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
slf4j-api/1.7.30//slf4j-api-1.7.30.jar
slf4j-log4j12/1.7.30//slf4j-log4j12-1.7.30.jar
snakeyaml/1.24//snakeyaml-1.24.jar
snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
Expand Down
8 changes: 4 additions & 4 deletions dev/deps/spark-deps-hadoop-3.2-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ javolution/5.5.1//javolution-5.5.1.jar
jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcip-annotations/1.0-1//jcip-annotations-1.0-1.jar
jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
jcl-over-slf4j/1.7.30//jcl-over-slf4j-1.7.30.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
jersey-client/2.30//jersey-client-2.30.jar
jersey-common/2.30//jersey-common-2.30.jar
Expand All @@ -148,7 +148,7 @@ json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
jsp-api/2.1//jsp-api-2.1.jar
jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kerb-admin/1.0.1//kerb-admin-1.0.1.jar
kerb-client/1.0.1//kerb-client-1.0.1.jar
kerb-common/1.0.1//kerb-common-1.0.1.jar
Expand Down Expand Up @@ -214,8 +214,8 @@ scala-reflect/2.12.10//scala-reflect-2.12.10.jar
scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
slf4j-api/1.7.30//slf4j-api-1.7.30.jar
slf4j-log4j12/1.7.30//slf4j-log4j12-1.7.30.jar
snakeyaml/1.24//snakeyaml-1.24.jar
snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
Expand Down
20 changes: 10 additions & 10 deletions docs/_data/menu-sql.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -156,22 +156,22 @@
url: sql-ref-syntax-qry-select-distribute-by.html
- text: LIMIT Clause
url: sql-ref-syntax-qry-select-limit.html
- text: Common Table Expression
url: sql-ref-syntax-qry-select-cte.html
- text: Inline Table
url: sql-ref-syntax-qry-select-inline-table.html
- text: JOIN
url: sql-ref-syntax-qry-select-join.html
- text: Join Hints
url: sql-ref-syntax-qry-select-hints.html
- text: LIKE Predicate
url: sql-ref-syntax-qry-select-like.html
- text: Set Operators
url: sql-ref-syntax-qry-select-setops.html
- text: TABLESAMPLE
url: sql-ref-syntax-qry-sampling.html
- text: Table-valued Function
url: sql-ref-syntax-qry-select-tvf.html
- text: Inline Table
url: sql-ref-syntax-qry-select-inline-table.html
- text: Common Table Expression
url: sql-ref-syntax-qry-select-cte.html
- text: LIKE Predicate
url: sql-ref-syntax-qry-select-like.html
- text: Window Function
url: sql-ref-syntax-qry-window.html
- text: EXPLAIN
Expand Down Expand Up @@ -213,20 +213,20 @@
subitems:
- text: SHOW COLUMNS
url: sql-ref-syntax-aux-show-columns.html
- text: SHOW CREATE TABLE
url: sql-ref-syntax-aux-show-create-table.html
- text: SHOW DATABASES
url: sql-ref-syntax-aux-show-databases.html
- text: SHOW FUNCTIONS
url: sql-ref-syntax-aux-show-functions.html
- text: SHOW PARTITIONS
url: sql-ref-syntax-aux-show-partitions.html
- text: SHOW TABLE
url: sql-ref-syntax-aux-show-table.html
- text: SHOW TABLES
url: sql-ref-syntax-aux-show-tables.html
- text: SHOW TBLPROPERTIES
url: sql-ref-syntax-aux-show-tblproperties.html
- text: SHOW PARTITIONS
url: sql-ref-syntax-aux-show-partitions.html
- text: SHOW CREATE TABLE
url: sql-ref-syntax-aux-show-create-table.html
- text: SHOW VIEWS
url: sql-ref-syntax-aux-show-views.html
- text: CONFIGURATION MANAGEMENT
Expand Down
8 changes: 4 additions & 4 deletions docs/sql-ref-ansi-compliance.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ This means that in case an operation causes overflows, the result is the same wi
On the other hand, Spark SQL returns null for decimal overflows.
When `spark.sql.ansi.enabled` is set to `true` and an overflow occurs in numeric and interval arithmetic operations, it throws an arithmetic exception at runtime.

{% highlight sql %}
```sql
-- `spark.sql.ansi.enabled=true`
SELECT 2147483647 + 1;
java.lang.ArithmeticException: integer overflow
Expand All @@ -78,7 +78,7 @@ SELECT 2147483647 + 1;
+----------------+
| -2147483648|
+----------------+
{% endhighlight %}
```

### Type Conversion

Expand All @@ -89,7 +89,7 @@ On the other hand, `INSERT INTO` syntax throws an analysis exception when the AN
Currently, the ANSI mode affects explicit casting and assignment casting only.
In future releases, the behaviour of type coercion might change along with the other two type conversion rules.

{% highlight sql %}
```sql
-- Examples of explicit casting

-- `spark.sql.ansi.enabled=true`
Expand Down Expand Up @@ -130,7 +130,7 @@ SELECT * FROM t;
+---+
| 1|
+---+
{% endhighlight %}
```

### SQL Functions

Expand Down
4 changes: 2 additions & 2 deletions docs/sql-ref-datatypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,7 +751,7 @@ Specifically:

#### Examples

{% highlight sql %}
```sql
SELECT double('infinity') AS col;
+--------+
| col|
Expand Down Expand Up @@ -824,4 +824,4 @@ SELECT COUNT(*), c2 FROM test GROUP BY c2;
| 2|-Infinity|
| 3| Infinity|
+---------+---------+
{% endhighlight %}
```
101 changes: 45 additions & 56 deletions docs/sql-ref-functions-udf-aggregate.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,46 +27,35 @@ User-Defined Aggregate Functions (UDAFs) are user-programmable routines that act

A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value.

* IN - The input type for the aggregation.
* BUF - The type of the intermediate value of the reduction.
* OUT - The type of the final output result.
***IN*** - The input type for the aggregation.

***BUF*** - The type of the intermediate value of the reduction.

***OUT*** - The type of the final output result.

* **bufferEncoder: Encoder[BUF]**

<dl>
<dt><code><em>bufferEncoder: Encoder[BUF]</em></code></dt>
<dd>
Specifies the Encoder for the intermediate value type.
</dd>
</dl>
<dl>
<dt><code><em>finish(reduction: BUF): OUT</em></code></dt>
<dd>

* **finish(reduction: BUF): OUT**

Transform the output of the reduction.
</dd>
</dl>
<dl>
<dt><code><em>merge(b1: BUF, b2: BUF): BUF</em></code></dt>
<dd>

* **merge(b1: BUF, b2: BUF): BUF**

Merge two intermediate values.
</dd>
</dl>
<dl>
<dt><code><em>outputEncoder: Encoder[OUT]</em></code></dt>
<dd>

* **outputEncoder: Encoder[OUT]**

Specifies the Encoder for the final output value type.
</dd>
</dl>
<dl>
<dt><code><em>reduce(b: BUF, a: IN): BUF</em></code></dt>
<dd>
Aggregate input value <code>a</code> into current intermediate value. For performance, the function may modify <code>b</code> and return it instead of constructing new object for <code>b</code>.
</dd>
</dl>
<dl>
<dt><code><em>zero: BUF</em></code></dt>
<dd>

* **reduce(b: BUF, a: IN): BUF**

Aggregate input value `a` into current intermediate value. For performance, the function may modify `b` and return it instead of constructing new object for `b`.

* **zero: BUF**

The initial value of the intermediate result for this aggregation.
</dd>
</dl>

### Examples

Expand Down Expand Up @@ -95,16 +84,16 @@ For example, a user-defined average for untyped DataFrames can look like:
{% include_example untyped_custom_aggregation java/org/apache/spark/examples/sql/JavaUserDefinedUntypedAggregation.java%}
</div>
<div data-lang="SQL" markdown="1">
{% highlight sql %}
```sql
-- Compile and place UDAF MyAverage in a JAR file called `MyAverage.jar` in /tmp.
CREATE FUNCTION myAverage AS 'MyAverage' USING JAR '/tmp/MyAverage.jar';

SHOW USER FUNCTIONS;
-- +------------------+
-- | function|
-- +------------------+
-- | default.myAverage|
-- +------------------+
+------------------+
| function|
+------------------+
| default.myAverage|
+------------------+

CREATE TEMPORARY VIEW employees
USING org.apache.spark.sql.json
Expand All @@ -113,26 +102,26 @@ OPTIONS (
);

SELECT * FROM employees;
-- +-------+------+
-- | name|salary|
-- +-------+------+
-- |Michael| 3000|
-- | Andy| 4500|
-- | Justin| 3500|
-- | Berta| 4000|
-- +-------+------+
+-------+------+
| name|salary|
+-------+------+
|Michael| 3000|
| Andy| 4500|
| Justin| 3500|
| Berta| 4000|
+-------+------+

SELECT myAverage(salary) as average_salary FROM employees;
-- +--------------+
-- |average_salary|
-- +--------------+
-- | 3750.0|
-- +--------------+
{% endhighlight %}
+--------------+
|average_salary|
+--------------+
| 3750.0|
+--------------+
```
</div>
</div>

### Related Statements

* [Scalar User Defined Functions (UDFs)](sql-ref-functions-udf-scalar.html)
* [Integration with Hive UDFs/UDAFs/UDTFs](sql-ref-functions-udf-hive.html)
* [Scalar User Defined Functions (UDFs)](sql-ref-functions-udf-scalar.html)
* [Integration with Hive UDFs/UDAFs/UDTFs](sql-ref-functions-udf-hive.html)
12 changes: 6 additions & 6 deletions docs/sql-ref-functions-udf-hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Spark SQL supports integration of Hive UDFs, UDAFs and UDTFs. Similar to Spark U
Hive has two UDF interfaces: [UDF](https://github.com/apache/hive/blob/master/udf/src/java/org/apache/hadoop/hive/ql/exec/UDF.java) and [GenericUDF](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java).
An example below uses [GenericUDFAbs](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAbs.java) derived from `GenericUDF`.

{% highlight sql %}
```sql
-- Register `GenericUDFAbs` and use it in Spark SQL.
-- Note that, if you use your own programmed one, you need to add a JAR containig it
-- into a classpath,
Expand All @@ -52,12 +52,12 @@ SELECT testUDF(value) FROM t;
| 2.0|
| 3.0|
+--------------+
{% endhighlight %}
```


An example below uses [GenericUDTFExplode](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java) derived from [GenericUDTF](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java).

{% highlight sql %}
```sql
-- Register `GenericUDTFExplode` and use it in Spark SQL
CREATE TEMPORARY FUNCTION hiveUDTF
AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode';
Expand All @@ -79,12 +79,12 @@ SELECT hiveUDTF(value) FROM t;
| 3|
| 4|
+---+
{% endhighlight %}
```

Hive has two UDAF interfaces: [UDAF](https://github.com/apache/hive/blob/master/udf/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java) and [GenericUDAFResolver](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFResolver.java).
An example below uses [GenericUDAFSum](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java) derived from `GenericUDAFResolver`.

{% highlight sql %}
```sql
-- Register `GenericUDAFSum` and use it in Spark SQL
CREATE TEMPORARY FUNCTION hiveUDAF
AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFSum';
Expand All @@ -105,4 +105,4 @@ SELECT key, hiveUDAF(value) FROM t GROUP BY key;
| b| 3|
| a| 3|
+---+---------------+
{% endhighlight %}
```
Loading