Skip to content

[SPARK-31612][SQL][DOCS][FOLLOW-UP] SQL Reference clean up #28428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-31612][SQL][DOCS][FOLLOW-UP] SQL Reference clean up #28428

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d16b5cc
[SPARK-31612][SQL][DOCS][FOLLOW-UP] SQL Reference clean up
huaxingao May 1, 2020
abaf962
fix
huaxingao May 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions docs/sql-ref-functions-udf-aggregate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,34 +100,34 @@ For example, a user-defined average for untyped DataFrames can look like:
CREATE FUNCTION myAverage AS 'MyAverage' USING JAR '/tmp/MyAverage.jar';

SHOW USER FUNCTIONS;
-- +------------------+
-- | function|
-- +------------------+
-- | default.myAverage|
-- +------------------+
+------------------+
| function|
+------------------+
| default.myAverage|
+------------------+

CREATE TEMPORARY VIEW employees
CREATE TEMPORARY VIEW employees;
USING org.apache.spark.sql.json
OPTIONS (
path "examples/src/main/resources/employees.json"
);

SELECT * FROM employees;
-- +-------+------+
-- | name|salary|
-- +-------+------+
-- |Michael| 3000|
-- | Andy| 4500|
-- | Justin| 3500|
-- | Berta| 4000|
-- +-------+------+

SELECT myAverage(salary) as average_salary FROM employees;
-- +--------------+
-- |average_salary|
-- +--------------+
-- | 3750.0|
-- +--------------+
+-------+------+
| name|salary|
+-------+------+
|Michael| 3000|
| Andy| 4500|
| Justin| 3500|
| Berta| 4000|
+-------+------+

SELECT myAverage(salary) AS average_salary FROM employees;
+--------------+
|average_salary|
+--------------+
| 3750.0|
+--------------+
{% endhighlight %}
</div>
</div>
Expand Down
2 changes: 2 additions & 0 deletions docs/sql-ref-identifier.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Note: If `spark.sql.ansi.enabled` is set to true, ANSI SQL reserved keywords can
{% highlight sql %}
-- This CREATE TABLE fails with ParseException because of the illegal identifier name a.b
CREATE TABLE test (a.b int);
-- output
Copy link
Member

@maropu maropu May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment looks fine, but I think we need the same comment in the other error output.

Copy link
Contributor Author

@huaxingao huaxingao May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe have the same format as ansi compliance page?
Screen Shot 2020-05-01 at 2 57 21 PM

Copy link
Member

@maropu maropu May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding -- output there looks okay, WDYT?

INSERT INTO t VALUES ('1');
-- output
  org.apache.spark.sql.AnalysisException: Cannot write incompatible data to table '`default`.`t`':
  - Cannot safely cast 'v': StringType to IntegerType;

Could you check documents in the other DBMS-like systems? As the others suggested, I think its better to follow the other document formats.

Copy link
Contributor Author

@huaxingao huaxingao May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I googled, but didn't have much luck. Only found this one at https://docs.snowflake.com/en/sql-reference/functions/validate.html

Screen Shot 2020-05-01 at 4 08 33 PM

I personally like to either put -- output or indent 2 spaces for the error message, or both (the way you suggested)

Copy link
Member

@maropu maropu May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @HyukjinKwon @dongjoon-hyun

org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE TABLE test (a.'(line 1, pos 20)

Expand All @@ -72,6 +73,7 @@ CREATE TABLE test (`a.b` int);

-- This CREATE TABLE fails with ParseException because special character ` is not escaped
CREATE TABLE test1 (`a`b` int);
-- output
org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE TABLE test (`a`b`'(line 1, pos 23)

Expand Down
1 change: 0 additions & 1 deletion docs/sql-ref-syntax-aux-cache-refresh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ REFRESH resource_path

{% highlight sql %}
-- The Path is resolved using the datasource's File Index.

CREATE TABLE test(ID INT) using parquet;
INSERT INTO test SELECT 1000;
CACHE TABLE test;
Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-aux-describe-query.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ describe the query output.
CREATE TABLE person (name STRING , age INT COMMENT 'Age column', address STRING);

-- Returns column metadata information for a simple select query
DESCRIBE QUERY select age, sum(age) FROM person GROUP BY age;
DESCRIBE QUERY SELECT age, sum(age) FROM person GROUP BY age;
+--------+---------+----------+
|col_name|data_type| comment|
+--------+---------+----------+
Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-aux-show-columns.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ license: |

### Description

Return the list of columns in a table. If the table does not exist, an exception is thrown.
Returns the list of columns in a table. If the table does not exist, an exception is thrown.

### Syntax

Expand Down
13 changes: 7 additions & 6 deletions docs/sql-ref-syntax-aux-show-databases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ license: |

### Description

Lists the databases that match an optionally supplied string pattern. If no
Lists the databases that match an optionally supplied regular expression pattern. If no
pattern is supplied then the command lists all the databases in the system.
Please note that the usage of `SCHEMAS` and `DATABASES` are interchangeable
and mean the same thing.
Expand All @@ -39,11 +39,12 @@ SHOW { DATABASES | SCHEMAS } [ LIKE regex_pattern ]
<dd>
Specifies a regular expression pattern that is used to filter the results of the
statement.
<ul>
<li>Only <code>*</code> and <code>|</code> are allowed as wildcard pattern.</li>
<li>Excluding <code>*</code> and <code>|</code>, the remaining pattern follows the regular expression semantics.</li>
<li>The leading and trailing blanks are trimmed in the input pattern before processing. The pattern match is case-insensitive.</li>
</ul>
<ul>
<li>Except for <code>*</code> and <code>|</code> character, the pattern works like a regular expression.</li>
<li><code>*</code> alone matches 0 or more characters and <code>|</code> is used to separate multiple different regular expressions,
any of which can match. </li>
<li>The leading and trailing blanks are trimmed in the input pattern before processing. The pattern match is case-insensitive.</li>
</ul>
Copy link
Contributor Author

@huaxingao huaxingao May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to the same description of regex_pattern as the one used in SHOW TABLE EXTENDED, SHOW TABLES, SHOW VIEWS.

</dd>
</dl>

Expand Down
13 changes: 7 additions & 6 deletions docs/sql-ref-syntax-aux-show-functions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ clause is optional and supported only for compatibility with other systems.
### Syntax

{% highlight sql %}
SHOW [ function_kind ] FUNCTIONS ( [ LIKE ] function_name | regex_pattern )
SHOW [ function_kind ] FUNCTIONS { [ LIKE ] function_name | regex_pattern }
{% endhighlight %}

### Parameters
Expand Down Expand Up @@ -60,11 +60,12 @@ SHOW [ function_kind ] FUNCTIONS ( [ LIKE ] function_name | regex_pattern )
<dd>
Specifies a regular expression pattern that is used to filter the results of the
statement.
<ul>
<li>Only <code>*</code> and <code>|</code> are allowed as wildcard pattern.</li>
<li>Excluding <code>*</code> and <code>|</code>, the remaining pattern follows the regular expression semantics.</li>
<li>The leading and trailing blanks are trimmed in the input pattern before processing. The pattern match is case-insensitive.</li>
</ul>
<ul>
<li>Except for <code>*</code> and <code>|</code> character, the pattern works like a regular expression.</li>
<li><code>*</code> alone matches 0 or more characters and <code>|</code> is used to separate multiple different regular expressions,
any of which can match. </li>
<li>The leading and trailing blanks are trimmed in the input pattern before processing. The pattern match is case-insensitive.</li>
</ul>
Copy link
Contributor Author

@huaxingao huaxingao May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to the same description of regex_pattern as the one used in SHOW TABLE EXTENDED, SHOW TABLES, SHOW VIEWS.

</dd>
</dl>

Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-aux-show-partitions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ SHOW PARTITIONS table_identifier [ partition_spec ]
for partitions. When specified, the partitions that match the partition spec are returned.<br><br>
<b>Syntax:</b>
<code>
PARTITION ( partition_col_name [ = partition_col_val ] [ , ... ] )
PARTITION ( partition_col_name = partition_col_val [ , ... ] )
</code>
</dd>
</dl>
Expand Down
8 changes: 4 additions & 4 deletions docs/sql-ref-syntax-aux-show-table.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ cannot be used with a partition specification.
### Syntax

{% highlight sql %}
SHOW TABLE EXTENDED [ IN | FROM database_name ] LIKE regex_pattern
SHOW TABLE EXTENDED [ { IN | FROM } database_name ] LIKE regex_pattern
[ partition_spec ]
{% endhighlight %}

Expand All @@ -60,7 +60,7 @@ SHOW TABLE EXTENDED [ IN | FROM database_name ] LIKE regex_pattern
for partitions. Note that a table regex cannot be used with a partition specification.<br><br>
<b>Syntax:</b>
<code>
PARTITION ( partition_col_name [ = partition_col_val ] [ , ... ] )
PARTITION ( partition_col_name = partition_col_val [ , ... ] )
</code>
</dd>
</dl>
Expand Down Expand Up @@ -152,7 +152,7 @@ SHOW TABLE EXTENDED LIKE `employe*`;
+--------+---------+----------+---------------------------------------------------------------+

-- show partition file system details
SHOW TABLE EXTENDED IN `default` LIKE `employee` PARTITION (`grade=1`);
SHOW TABLE EXTENDED IN default LIKE `employee` PARTITION (`grade=1`);
+--------+---------+-----------+--------------------------------------------------------------+
|database|tableName|isTemporary| information |
+--------+---------+-----------+--------------------------------------------------------------+
Expand All @@ -175,7 +175,7 @@ SHOW TABLE EXTENDED IN `default` LIKE `employee` PARTITION (`grade=1`);
+--------+---------+-----------+--------------------------------------------------------------+

-- show partition file system details with regex fails as shown below
SHOW TABLE EXTENDED IN `default` LIKE `empl*` PARTITION (`grade=1`);
SHOW TABLE EXTENDED IN default LIKE `empl*` PARTITION (`grade=1`);
Error: Error running query: org.apache.spark.sql.catalyst.analysis.NoSuchTableException:
Table or view 'emplo*' not found in database 'default'; (state=,code=0)
{% endhighlight %}
Expand Down
4 changes: 2 additions & 2 deletions docs/sql-ref-syntax-aux-show-views.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ USE userdb;
CREATE VIEW user1 AS SELECT id, salary FROM default.employee WHERE name = 'user1';
CREATE VIEW user2 AS SELECT id, salary FROM default.employee WHERE name = 'user2';
USE default;
CREATE GLOBAL TEMP VIEW temp1 AS SELECT 1 as col1;
CREATE TEMP VIEW temp2 AS SELECT 1 as col1;
CREATE GLOBAL TEMP VIEW temp1 AS SELECT 1 AS col1;
CREATE TEMP VIEW temp2 AS SELECT 1 AS col1;

-- List all views in default database
SHOW VIEWS;
Expand Down
8 changes: 4 additions & 4 deletions docs/sql-ref-syntax-qry-sampling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ Note: `TABLESAMPLE` returns the approximate number of rows or fraction requested
### Syntax

{% highlight sql %}
TABLESAMPLE ((integer_expression | decimal_expression) PERCENT)
| TABLESAMPLE (integer_expression ROWS)
| TABLESAMPLE (BUCKET integer_expression OUT OF integer_expression)
TABLESAMPLE ( { integer_expression | decimal_expression } PERCENT )
| TABLESAMPLE ( integer_expression ROWS )
| TABLESAMPLE ( BUCKET integer_expression OUT OF integer_expression )
{% endhighlight %}

### Examples
Expand Down Expand Up @@ -89,6 +89,6 @@ SELECT * FROM test TABLESAMPLE (BUCKET 4 OUT OF 10);
+--+----+
{% endhighlight %}

### Related Statement
### Related Statements

* [SELECT](sql-ref-syntax-qry-select.html)
4 changes: 2 additions & 2 deletions docs/sql-ref-syntax-qry-select-cte.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ WITH common_table_expression [ , ... ]

While `common_table_expression` is defined as
{% highlight sql %}
expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( [ common_table_expression ] query )
expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( query )
{% endhighlight %}

### Parameters
Expand Down Expand Up @@ -62,7 +62,7 @@ SELECT * FROM t WHERE x = 1 AND y = 2;
+---+---+

-- CTE in CTE definition
WITH t as (
WITH t AS (
WITH t2 AS (SELECT 1)
SELECT * FROM t2
)
Expand Down
8 changes: 7 additions & 1 deletion docs/sql-ref-syntax-qry-select-hints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ license: |

Join Hints allow users to suggest the join strategy that Spark should use. Prior to Spark 3.0, only the `BROADCAST` Join Hint was supported. `MERGE`, `SHUFFLE_HASH` and `SHUFFLE_REPLICATE_NL` Joint Hints support was added in 3.0. When different join strategy hints are specified on both sides of a join, Spark prioritizes hints in the following order: `BROADCAST` over `MERGE` over `SHUFFLE_HASH` over `SHUFFLE_REPLICATE_NL`. When both sides are specified with the `BROADCAST` hint or the `SHUFFLE_HASH` hint, Spark will pick the build side based on the join type and the sizes of the relations. Since a given strategy may not support all join types, Spark is not guaranteed to use the join strategy suggested by the hint.

### Syntax

{% highlight sql %}
/*+ join_hint [ , ... ] */
{% endhighlight %}

### Join Hints Types

<dl>
Expand Down Expand Up @@ -78,7 +84,7 @@ SELECT /*+ SHUFFLE_REPLICATE_NL(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.ke
-- Spark will issue Warning in the following example
-- org.apache.spark.sql.catalyst.analysis.HintErrorLogger: Hint (strategy=merge)
-- is overridden by another hint and will not take effect.
SELECT /*+ BROADCAST(t1) */ /*+ MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ BROADCAST(t1), MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
Copy link
Contributor Author

@huaxingao huaxingao May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SELECT /*+ BROADCAST(t1) */ /*+ MERGE(t1, t2) */ works too, but want to make the example consistent with the syntax /*+ join_hint [ , ... ] */

{% endhighlight %}

### Related Statements
Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-qry-select-inline-table.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,6 @@ SELECT * FROM VALUES ("one", array(0, 1)), ("two", array(2, 3)) AS data(a, b);
+---+------+
{% endhighlight %}

### Related Statement
### Related Statements

* [SELECT](sql-ref-syntax-qry-select.html)
3 changes: 1 addition & 2 deletions docs/sql-ref-syntax-qry-select-join.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ relation { [ join_type ] JOIN relation [ join_criteria ] | NATURAL join_type JOI
Specifies how the rows from one relation will be combined with the rows of another relation.<br><br>
<b>Syntax:</b>
<code>
ON boolean_expression | USING ( column_name [ , column_name ... ] )
ON boolean_expression | USING ( column_name [ , ... ] )
</code> <br><br>
<code>boolean_expression</code><br>
Specifies an expression with a return type of boolean.
Expand Down Expand Up @@ -139,7 +139,6 @@ An anti join returns values from the left relation that has no match with the ri
{% highlight sql %}
-- Use employee and department tables to demonstrate different type of joins.
SELECT * FROM employee;

+---+-----+------+
| id| name|deptno|
+---+-----+------+
Expand Down
3 changes: 1 addition & 2 deletions docs/sql-ref-syntax-qry-select-like.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,11 @@ SELECT * FROM person WHERE name NOT LIKE 'M_ry';
|400| Dan| 50|
+---+------+---+

SELECT * FROM person WHERE name RLIKE '[MD]';
SELECT * FROM person WHERE name RLIKE 'M+';
+---+----+----+
| id|name| age|
+---+----+----+
|300|Mike| 80|
|400| Dan| 50|
|200|Mary|null|
+---+----+----+

Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-qry-select-tvf.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,6 @@ SELECT * FROM range(5, 8) AS test;
+---+
{% endhighlight %}

### Related Statement
### Related Statements

* [SELECT](sql-ref-syntax-qry-select.html)
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-qry-window.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ window_function OVER
MAX | MIN | COUNT | SUM | AVG | ...
</code>
<br>
Please refer to the <a href="api/sql/">Built-in Functions</a> document for a complete list of Spark aggregate functions.
Please refer to the <a href="sql-ref-functions-builtin.html#aggregate-functions">Built-in Aggregation Functions</a> document for a complete list of Spark aggregate functions.
</ul>
</dd>
</dl>
Expand Down
1 change: 1 addition & 0 deletions docs/sql-ref-syntax-qry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,6 @@ ability to generate logical and physical plan for a given query using
* [Table-valued Function](sql-ref-syntax-qry-select-tvf.html)
* [Inline Table](sql-ref-syntax-qry-select-inline-table.html)
* [Common Table Expression](sql-ref-syntax-qry-select-cte.html)
* [LIKE Predicate](sql-ref-syntax-qry-select-like.html)
* [Window Function](sql-ref-syntax-qry-window.html)
* [EXPLAIN Statement](sql-ref-syntax-qry-explain.html)