diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md index 3a0c2c123d81..259dd977fe99 100644 --- a/docs/sql-ref-ansi-compliance.md +++ b/docs/sql-ref-ansi-compliance.md @@ -27,35 +27,10 @@ The casting behaviours are defined as store assignment rules in the standard. When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. This is a separate configuration because its default value is `ANSI`, while the configuration `spark.sql.ansi.enabled` is disabled by default. - - - - - - - - - - - - - - -
Property NameDefaultMeaningSince Version
spark.sql.ansi.enabledfalse - (Experimental) When true, Spark tries to conform to the ANSI SQL specification: - 1. Spark will throw a runtime exception if an overflow occurs in any operation on integral/decimal field. - 2. Spark will forbid using the reserved keywords of ANSI SQL as identifiers in the SQL parser. - 3.0.0
spark.sql.storeAssignmentPolicyANSI - (Experimental) When inserting a value into a column with different data type, Spark will perform type coercion. - Currently, we support 3 policies for the type coercion rules: ANSI, legacy and strict. With ANSI policy, - Spark performs the type coercion as per ANSI SQL. In practice, the behavior is mostly the same as PostgreSQL. - It disallows certain unreasonable type conversions such as converting string to int or double to boolean. - With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose. - e.g. converting string to int or double to boolean is allowed. - It is also the only behavior in Spark 2.x and it is compatible with Hive. - With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion, - e.g. converting double to int or decimal to double is not allowed. - 3.0.0
+|Property Name|Default|Meaning|Since Version| +|-------------|-------|-------|-------------| +|`spark.sql.ansi.enabled`|false|(Experimental) When true, Spark tries to conform to the ANSI SQL specification:
1. Spark will throw a runtime exception if an overflow occurs in any operation on integral/decimal field.
2. Spark will forbid using the reserved keywords of ANSI SQL as identifiers in the SQL parser.|3.0.0| +|`spark.sql.storeAssignmentPolicy`|ANSI|(Experimental) When inserting a value into a column with different data type, Spark will perform type coercion. Currently, we support 3 policies for the type coercion rules: ANSI, legacy and strict. With ANSI policy, Spark performs the type coercion as per ANSI SQL. In practice, the behavior is mostly the same as PostgreSQL. It disallows certain unreasonable type conversions such as converting string to int or double to boolean. With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose. e.g. converting string to int or double to boolean is allowed. It is also the only behavior in Spark 2.x and it is compatible with Hive. With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion, e.g. converting double to int or decimal to double is not allowed.|3.0.0| The following subsections present behaviour changes in arithmetic operations, type conversions, and SQL parsing when the ANSI mode enabled. @@ -152,259 +127,256 @@ By default `spark.sql.ansi.enabled` is false. Below is a list of all the keywords in Spark SQL. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
KeywordSpark SQLSQL-2011
ANSI modedefault mode
ADDnon-reservednon-reservednon-reserved
AFTERnon-reservednon-reservednon-reserved
ALLreservednon-reservedreserved
ALTERnon-reservednon-reservedreserved
ANALYZEnon-reservednon-reservednon-reserved
ANDreservednon-reservedreserved
ANTIreservedstrict-non-reservednon-reserved
ANYreservednon-reservedreserved
ARCHIVEnon-reservednon-reservednon-reserved
ARRAYnon-reservednon-reservedreserved
ASreservednon-reservedreserved
ASCnon-reservednon-reservednon-reserved
ATnon-reservednon-reservedreserved
AUTHORIZATIONreservednon-reservedreserved
BETWEENnon-reservednon-reservedreserved
BOTHreservednon-reservedreserved
BUCKETnon-reservednon-reservednon-reserved
BUCKETSnon-reservednon-reservednon-reserved
BYnon-reservednon-reservedreserved
CACHEnon-reservednon-reservednon-reserved
CASCADEnon-reservednon-reservedreserved
CASEreservednon-reservedreserved
CASTreservednon-reservedreserved
CHANGEnon-reservednon-reservednon-reserved
CHECKreservednon-reservedreserved
CLEARnon-reservednon-reservednon-reserved
CLUSTERnon-reservednon-reservednon-reserved
CLUSTEREDnon-reservednon-reservednon-reserved
CODEGENnon-reservednon-reservednon-reserved
COLLATEreservednon-reservedreserved
COLLECTIONnon-reservednon-reservednon-reserved
COLUMNreservednon-reservedreserved
COLUMNSnon-reservednon-reservednon-reserved
COMMENTnon-reservednon-reservednon-reserved
COMMITnon-reservednon-reservedreserved
COMPACTnon-reservednon-reservednon-reserved
COMPACTIONSnon-reservednon-reservednon-reserved
COMPUTEnon-reservednon-reservednon-reserved
CONCATENATEnon-reservednon-reservednon-reserved
CONSTRAINTreservednon-reservedreserved
COSTnon-reservednon-reservednon-reserved
CREATEreservednon-reservedreserved
CROSSreservedstrict-non-reservedreserved
CUBEnon-reservednon-reservedreserved
CURRENTnon-reservednon-reservedreserved
CURRENT_DATEreservednon-reservedreserved
CURRENT_TIMEreservednon-reservedreserved
CURRENT_TIMESTAMPreservednon-reservedreserved
CURRENT_USERreservednon-reservedreserved
DATAnon-reservednon-reservednon-reserved
DATABASEnon-reservednon-reservednon-reserved
DATABASESnon-reservednon-reservednon-reserved
DAYreservednon-reservedreserved
DBPROPERTIESnon-reservednon-reservednon-reserved
DEFINEDnon-reservednon-reservednon-reserved
DELETEnon-reservednon-reservedreserved
DELIMITEDnon-reservednon-reservednon-reserved
DESCnon-reservednon-reservednon-reserved
DESCRIBEnon-reservednon-reservedreserved
DFSnon-reservednon-reservednon-reserved
DIRECTORIESnon-reservednon-reservednon-reserved
DIRECTORYnon-reservednon-reservednon-reserved
DISTINCTreservednon-reservedreserved
DISTRIBUTEnon-reservednon-reservednon-reserved
DIVnon-reservednon-reservednon-reserved
DROPnon-reservednon-reservedreserved
ELSEreservednon-reservedreserved
ENDreservednon-reservedreserved
ESCAPEreservednon-reservedreserved
ESCAPEDnon-reservednon-reservednon-reserved
EXCEPTreservedstrict-non-reservedreserved
EXCHANGEnon-reservednon-reservednon-reserved
EXISTSnon-reservednon-reservedreserved
EXPLAINnon-reservednon-reservednon-reserved
EXPORTnon-reservednon-reservednon-reserved
EXTENDEDnon-reservednon-reservednon-reserved
EXTERNALnon-reservednon-reservedreserved
EXTRACTnon-reservednon-reservedreserved
FALSEreservednon-reservedreserved
FETCHreservednon-reservedreserved
FIELDSnon-reservednon-reservednon-reserved
FILTERreservednon-reservedreserved
FILEFORMATnon-reservednon-reservednon-reserved
FIRSTnon-reservednon-reservednon-reserved
FOLLOWINGnon-reservednon-reservednon-reserved
FORreservednon-reservedreserved
FOREIGNreservednon-reservedreserved
FORMATnon-reservednon-reservednon-reserved
FORMATTEDnon-reservednon-reservednon-reserved
FROMreservednon-reservedreserved
FULLreservedstrict-non-reservedreserved
FUNCTIONnon-reservednon-reservedreserved
FUNCTIONSnon-reservednon-reservednon-reserved
GLOBALnon-reservednon-reservedreserved
GRANTreservednon-reservedreserved
GROUPreservednon-reservedreserved
GROUPINGnon-reservednon-reservedreserved
HAVINGreservednon-reservedreserved
HOURreservednon-reservedreserved
IFnon-reservednon-reservedreserved
IGNOREnon-reservednon-reservednon-reserved
IMPORTnon-reservednon-reservednon-reserved
INreservednon-reservedreserved
INDEXnon-reservednon-reservednon-reserved
INDEXESnon-reservednon-reservednon-reserved
INNERreservedstrict-non-reservedreserved
INPATHnon-reservednon-reservednon-reserved
INPUTFORMATnon-reservednon-reservednon-reserved
INSERTnon-reservednon-reservedreserved
INTERSECTreservedstrict-non-reservedreserved
INTERVALnon-reservednon-reservedreserved
INTOreservednon-reservedreserved
ISreservednon-reservedreserved
ITEMSnon-reservednon-reservednon-reserved
JOINreservedstrict-non-reservedreserved
KEYSnon-reservednon-reservednon-reserved
LASTnon-reservednon-reservednon-reserved
LATERALnon-reservednon-reservedreserved
LAZYnon-reservednon-reservednon-reserved
LEADINGreservednon-reservedreserved
LEFTreservedstrict-non-reservedreserved
LIKEnon-reservednon-reservedreserved
LIMITnon-reservednon-reservednon-reserved
LINESnon-reservednon-reservednon-reserved
LISTnon-reservednon-reservednon-reserved
LOADnon-reservednon-reservednon-reserved
LOCALnon-reservednon-reservedreserved
LOCATIONnon-reservednon-reservednon-reserved
LOCKnon-reservednon-reservednon-reserved
LOCKSnon-reservednon-reservednon-reserved
LOGICALnon-reservednon-reservednon-reserved
MACROnon-reservednon-reservednon-reserved
MAPnon-reservednon-reservednon-reserved
MATCHEDnon-reservednon-reservednon-reserved
MERGEnon-reservednon-reservednon-reserved
MINUSreservedstrict-non-reservednon-reserved
MINUTEreservednon-reservedreserved
MONTHreservednon-reservedreserved
MSCKnon-reservednon-reservednon-reserved
NAMESPACEnon-reservednon-reservednon-reserved
NAMESPACESnon-reservednon-reservednon-reserved
NATURALreservedstrict-non-reservedreserved
NOnon-reservednon-reservedreserved
NOTreservednon-reservedreserved
NULLreservednon-reservedreserved
NULLSnon-reservednon-reservednon-reserved
OFnon-reservednon-reservedreserved
ONreservedstrict-non-reservedreserved
ONLYreservednon-reservedreserved
OPTIONnon-reservednon-reservednon-reserved
OPTIONSnon-reservednon-reservednon-reserved
ORreservednon-reservedreserved
ORDERreservednon-reservedreserved
OUTnon-reservednon-reservedreserved
OUTERreservednon-reservedreserved
OUTPUTFORMATnon-reservednon-reservednon-reserved
OVERnon-reservednon-reservednon-reserved
OVERLAPSreservednon-reservedreserved
OVERLAYnon-reservednon-reservednon-reserved
OVERWRITEnon-reservednon-reservednon-reserved
PARTITIONnon-reservednon-reservedreserved
PARTITIONEDnon-reservednon-reservednon-reserved
PARTITIONSnon-reservednon-reservednon-reserved
PERCENTnon-reservednon-reservednon-reserved
PIVOTnon-reservednon-reservednon-reserved
PLACINGnon-reservednon-reservednon-reserved
POSITIONnon-reservednon-reservedreserved
PRECEDINGnon-reservednon-reservednon-reserved
PRIMARYreservednon-reservedreserved
PRINCIPALSnon-reservednon-reservednon-reserved
PROPERTIESnon-reservednon-reservednon-reserved
PURGEnon-reservednon-reservednon-reserved
QUERYnon-reservednon-reservednon-reserved
RANGEnon-reservednon-reservedreserved
RECORDREADERnon-reservednon-reservednon-reserved
RECORDWRITERnon-reservednon-reservednon-reserved
RECOVERnon-reservednon-reservednon-reserved
REDUCEnon-reservednon-reservednon-reserved
REFERENCESreservednon-reservedreserved
REFRESHnon-reservednon-reservednon-reserved
RENAMEnon-reservednon-reservednon-reserved
REPAIRnon-reservednon-reservednon-reserved
REPLACEnon-reservednon-reservednon-reserved
RESETnon-reservednon-reservednon-reserved
RESTRICTnon-reservednon-reservednon-reserved
REVOKEnon-reservednon-reservedreserved
RIGHTreservedstrict-non-reservedreserved
RLIKEnon-reservednon-reservednon-reserved
ROLEnon-reservednon-reservednon-reserved
ROLESnon-reservednon-reservednon-reserved
ROLLBACKnon-reservednon-reservedreserved
ROLLUPnon-reservednon-reservedreserved
ROWnon-reservednon-reservedreserved
ROWSnon-reservednon-reservedreserved
SCHEMAnon-reservednon-reservednon-reserved
SECONDreservednon-reservedreserved
SELECTreservednon-reservedreserved
SEMIreservedstrict-non-reservednon-reserved
SEPARATEDnon-reservednon-reservednon-reserved
SERDEnon-reservednon-reservednon-reserved
SERDEPROPERTIESnon-reservednon-reservednon-reserved
SESSION_USERreservednon-reservedreserved
SETnon-reservednon-reservedreserved
SETSnon-reservednon-reservednon-reserved
SHOWnon-reservednon-reservednon-reserved
SKEWEDnon-reservednon-reservednon-reserved
SOMEreservednon-reservedreserved
SORTnon-reservednon-reservednon-reserved
SORTEDnon-reservednon-reservednon-reserved
STARTnon-reservednon-reservedreserved
STATISTICSnon-reservednon-reservednon-reserved
STOREDnon-reservednon-reservednon-reserved
STRATIFYnon-reservednon-reservednon-reserved
STRUCTnon-reservednon-reservednon-reserved
SUBSTRnon-reservednon-reservednon-reserved
SUBSTRINGnon-reservednon-reservednon-reserved
TABLEreservednon-reservedreserved
TABLESnon-reservednon-reservednon-reserved
TABLESAMPLEnon-reservednon-reservedreserved
TBLPROPERTIESnon-reservednon-reservednon-reserved
TEMPORARYnon-reservednon-reservednon-reserved
TERMINATEDnon-reservednon-reservednon-reserved
THENreservednon-reservedreserved
TOreservednon-reservedreserved
TOUCHnon-reservednon-reservednon-reserved
TRAILINGreservednon-reservedreserved
TRANSACTIONnon-reservednon-reservednon-reserved
TRANSACTIONSnon-reservednon-reservednon-reserved
TRANSFORMnon-reservednon-reservednon-reserved
TRIMnon-reservednon-reservednon-reserved
TRUEnon-reservednon-reservedreserved
TRUNCATEnon-reservednon-reservedreserved
UNARCHIVEnon-reservednon-reservednon-reserved
UNBOUNDEDnon-reservednon-reservednon-reserved
UNCACHEnon-reservednon-reservednon-reserved
UNIONreservedstrict-non-reservedreserved
UNIQUEreservednon-reservedreserved
UNKNOWNreservednon-reservedreserved
UNLOCKnon-reservednon-reservednon-reserved
UNSETnon-reservednon-reservednon-reserved
UPDATEnon-reservednon-reservedreserved
USEnon-reservednon-reservednon-reserved
USERreservednon-reservedreserved
USINGreservedstrict-non-reservedreserved
VALUESnon-reservednon-reservedreserved
VIEWnon-reservednon-reservednon-reserved
VIEWSnon-reservednon-reservednon-reserved
WHENreservednon-reservedreserved
WHEREreservednon-reservedreserved
WINDOWnon-reservednon-reservedreserved
WITHreservednon-reservedreserved
YEARreservednon-reservedreserved
+|Keyword|Spark SQL
ANSI Mode|Spark SQL
Default Mode|SQL-2011| +|-------|----------------------|-------------------------|--------| +|ADD|non-reserved|non-reserved|non-reserved| +|AFTER|non-reserved|non-reserved|non-reserved| +|ALL|reserved|non-reserved|reserved| +|ALTER|non-reserved|non-reserved|reserved| +|ANALYZE|non-reserved|non-reserved|non-reserved| +|AND|reserved|non-reserved|reserved| +|ANTI|reserved|strict-non-reserved|non-reserved| +|ANY|reserved|non-reserved|reserved| +|ARCHIVE|non-reserved|non-reserved|non-reserved| +|ARRAY|non-reserved|non-reserved|reserved| +|AS|reserved|non-reserved|reserved| +|ASC|non-reserved|non-reserved|non-reserved| +|AT|non-reserved|non-reserved|reserved| +|AUTHORIZATION|reserved|non-reserved|reserved| +|BETWEEN|non-reserved|non-reserved|reserved| +|BOTH|reserved|non-reserved|reserved| +|BUCKET|non-reserved|non-reserved|non-reserved| +|BUCKETS|non-reserved|non-reserved|non-reserved| +|BY|non-reserved|non-reserved|reserved| +|CACHE|non-reserved|non-reserved|non-reserved| +|CASCADE|non-reserved|non-reserved|reserved| +|CASE|reserved|non-reserved|reserved| +|CAST|reserved|non-reserved|reserved| +|CHANGE|non-reserved|non-reserved|non-reserved| +|CHECK|reserved|non-reserved|reserved| +|CLEAR|non-reserved|non-reserved|non-reserved| +|CLUSTER|non-reserved|non-reserved|non-reserved| +|CLUSTERED|non-reserved|non-reserved|non-reserved| +|CODEGEN|non-reserved|non-reserved|non-reserved| +|COLLATE|reserved|non-reserved|reserved| +|COLLECTION|non-reserved|non-reserved|non-reserved| +|COLUMN|reserved|non-reserved|reserved| +|COLUMNS|non-reserved|non-reserved|non-reserved| +|COMMENT|non-reserved|non-reserved|non-reserved| +|COMMIT|non-reserved|non-reserved|reserved| +|COMPACT|non-reserved|non-reserved|non-reserved| +|COMPACTIONS|non-reserved|non-reserved|non-reserved| +|COMPUTE|non-reserved|non-reserved|non-reserved| +|CONCATENATE|non-reserved|non-reserved|non-reserved| +|CONSTRAINT|reserved|non-reserved|reserved| +|COST|non-reserved|non-reserved|non-reserved| +|CREATE|reserved|non-reserved|reserved| +|CROSS|reserved|strict-non-reserved|reserved| +|CUBE|non-reserved|non-reserved|reserved| +|CURRENT|non-reserved|non-reserved|reserved| +|CURRENT_DATE|reserved|non-reserved|reserved| +|CURRENT_TIME|reserved|non-reserved|reserved| +|CURRENT_TIMESTAMP|reserved|non-reserved|reserved| +|CURRENT_USER|reserved|non-reserved|reserved| +|DATA|non-reserved|non-reserved|non-reserved| +|DATABASE|non-reserved|non-reserved|non-reserved| +|DATABASES|non-reserved|non-reserved|non-reserved| +|DAY|reserved|non-reserved|reserved| +|DBPROPERTIES|non-reserved|non-reserved|non-reserved| +|DEFINED|non-reserved|non-reserved|non-reserved| +|DELETE|non-reserved|non-reserved|reserved| +|DELIMITED|non-reserved|non-reserved|non-reserved| +|DESC|non-reserved|non-reserved|non-reserved| +|DESCRIBE|non-reserved|non-reserved|reserved| +|DFS|non-reserved|non-reserved|non-reserved| +|DIRECTORIES|non-reserved|non-reserved|non-reserved| +|DIRECTORY|non-reserved|non-reserved|non-reserved| +|DISTINCT|reserved|non-reserved|reserved| +|DISTRIBUTE|non-reserved|non-reserved|non-reserved| +|DIV|non-reserved|non-reserved|non-reserved| +|DROP|non-reserved|non-reserved|reserved| +|ELSE|reserved|non-reserved|reserved| +|END|reserved|non-reserved|reserved| +|ESCAPE|reserved|non-reserved|reserved| +|ESCAPED|non-reserved|non-reserved|non-reserved| +|EXCEPT|reserved|strict-non-reserved|reserved| +|EXCHANGE|non-reserved|non-reserved|non-reserved| +|EXISTS|non-reserved|non-reserved|reserved| +|EXPLAIN|non-reserved|non-reserved|non-reserved| +|EXPORT|non-reserved|non-reserved|non-reserved| +|EXTENDED|non-reserved|non-reserved|non-reserved| +|EXTERNAL|non-reserved|non-reserved|reserved| +|EXTRACT|non-reserved|non-reserved|reserved| +|FALSE|reserved|non-reserved|reserved| +|FETCH|reserved|non-reserved|reserved| +|FIELDS|non-reserved|non-reserved|non-reserved| +|FILTER|reserved|non-reserved|reserved| +|FILEFORMAT|non-reserved|non-reserved|non-reserved| +|FIRST|non-reserved|non-reserved|non-reserved| +|FOLLOWING|non-reserved|non-reserved|non-reserved| +|FOR|reserved|non-reserved|reserved| +|FOREIGN|reserved|non-reserved|reserved| +|FORMAT|non-reserved|non-reserved|non-reserved| +|FORMATTED|non-reserved|non-reserved|non-reserved| +|FROM|reserved|non-reserved|reserved| +|FULL|reserved|strict-non-reserved|reserved| +|FUNCTION|non-reserved|non-reserved|reserved| +|FUNCTIONS|non-reserved|non-reserved|non-reserved| +|GLOBAL|non-reserved|non-reserved|reserved| +|GRANT|reserved|non-reserved|reserved| +|GROUP|reserved|non-reserved|reserved| +|GROUPING|non-reserved|non-reserved|reserved| +|HAVING|reserved|non-reserved|reserved| +|HOUR|reserved|non-reserved|reserved| +|IF|non-reserved|non-reserved|reserved| +|IGNORE|non-reserved|non-reserved|non-reserved| +|IMPORT|non-reserved|non-reserved|non-reserved| +|IN|reserved|non-reserved|reserved| +|INDEX|non-reserved|non-reserved|non-reserved| +|INDEXES|non-reserved|non-reserved|non-reserved| +|INNER|reserved|strict-non-reserved|reserved| +|INPATH|non-reserved|non-reserved|non-reserved| +|INPUTFORMAT|non-reserved|non-reserved|non-reserved| +|INSERT|non-reserved|non-reserved|reserved| +|INTERSECT|reserved|strict-non-reserved|reserved| +|INTERVAL|non-reserved|non-reserved|reserved| +|INTO|reserved|non-reserved|reserved| +|IS|reserved|non-reserved|reserved| +|ITEMS|non-reserved|non-reserved|non-reserved| +|JOIN|reserved|strict-non-reserved|reserved| +|KEYS|non-reserved|non-reserved|non-reserved| +|LAST|non-reserved|non-reserved|non-reserved| +|LATERAL|non-reserved|non-reserved|reserved| +|LAZY|non-reserved|non-reserved|non-reserved| +|LEADING|reserved|non-reserved|reserved| +|LEFT|reserved|strict-non-reserved|reserved| +|LIKE|non-reserved|non-reserved|reserved| +|LIMIT|non-reserved|non-reserved|non-reserved| +|LINES|non-reserved|non-reserved|non-reserved| +|LIST|non-reserved|non-reserved|non-reserved| +|LOAD|non-reserved|non-reserved|non-reserved| +|LOCAL|non-reserved|non-reserved|reserved| +|LOCATION|non-reserved|non-reserved|non-reserved| +|LOCK|non-reserved|non-reserved|non-reserved| +|LOCKS|non-reserved|non-reserved|non-reserved| +|LOGICAL|non-reserved|non-reserved|non-reserved| +|MACRO|non-reserved|non-reserved|non-reserved| +|MAP|non-reserved|non-reserved|non-reserved| +|MATCHED|non-reserved|non-reserved|non-reserved| +|MERGE|non-reserved|non-reserved|non-reserved| +|MINUS|reserved|strict-non-reserved|non-reserved| +|MINUTE|reserved|non-reserved|reserved| +|MONTH|reserved|non-reserved|reserved| +|MSCK|non-reserved|non-reserved|non-reserved| +|NAMESPACE|non-reserved|non-reserved|non-reserved| +|NAMESPACES|non-reserved|non-reserved|non-reserved| +|NATURAL|reserved|strict-non-reserved|reserved| +|NO|non-reserved|non-reserved|reserved| +|NOT|reserved|non-reserved|reserved| +|NULL|reserved|non-reserved|reserved| +|NULLS|non-reserved|non-reserved|non-reserved| +|OF|non-reserved|non-reserved|reserved| +|ON|reserved|strict-non-reserved|reserved| +|ONLY|reserved|non-reserved|reserved| +|OPTION|non-reserved|non-reserved|non-reserved| +|OPTIONS|non-reserved|non-reserved|non-reserved| +|OR|reserved|non-reserved|reserved| +|ORDER|reserved|non-reserved|reserved| +|OUT|non-reserved|non-reserved|reserved| +|OUTER|reserved|non-reserved|reserved| +|OUTPUTFORMAT|non-reserved|non-reserved|non-reserved| +|OVER|non-reserved|non-reserved|non-reserved| +|OVERLAPS|reserved|non-reserved|reserved| +|OVERLAY|non-reserved|non-reserved|non-reserved| +|OVERWRITE|non-reserved|non-reserved|non-reserved| +|PARTITION|non-reserved|non-reserved|reserved| +|PARTITIONED|non-reserved|non-reserved|non-reserved| +|PARTITIONS|non-reserved|non-reserved|non-reserved| +|PERCENT|non-reserved|non-reserved|non-reserved| +|PIVOT|non-reserved|non-reserved|non-reserved| +|PLACING|non-reserved|non-reserved|non-reserved| +|POSITION|non-reserved|non-reserved|reserved| +|PRECEDING|non-reserved|non-reserved|non-reserved| +|PRIMARY|reserved|non-reserved|reserved| +|PRINCIPALS|non-reserved|non-reserved|non-reserved| +|PROPERTIES|non-reserved|non-reserved|non-reserved| +|PURGE|non-reserved|non-reserved|non-reserved| +|QUERY|non-reserved|non-reserved|non-reserved| +|RECORDREADER|non-reserved|non-reserved|non-reserved| +|RECORDWRITER|non-reserved|non-reserved|non-reserved| +|RECOVER|non-reserved|non-reserved|non-reserved| +|REDUCE|non-reserved|non-reserved|non-reserved| +|REFERENCES|reserved|non-reserved|reserved| +|REFRESH|non-reserved|non-reserved|non-reserved| +|RENAME|non-reserved|non-reserved|non-reserved| +|REPAIR|non-reserved|non-reserved|non-reserved| +|REPLACE|non-reserved|non-reserved|non-reserved| +|RESET|non-reserved|non-reserved|non-reserved| +|RESTRICT|non-reserved|non-reserved|non-reserved| +|REVOKE|non-reserved|non-reserved|reserved| +|RIGHT|reserved|strict-non-reserved|reserved| +|RLIKE|non-reserved|non-reserved|non-reserved| +|ROLE|non-reserved|non-reserved|non-reserved| +|ROLES|non-reserved|non-reserved|non-reserved| +|ROLLBACK|non-reserved|non-reserved|reserved| +|ROLLUP|non-reserved|non-reserved|reserved| +|ROW|non-reserved|non-reserved|reserved| +|ROWS|non-reserved|non-reserved|reserved| +|SCHEMA|non-reserved|non-reserved|non-reserved| +|SECOND|reserved|non-reserved|reserved| +|SELECT|reserved|non-reserved|reserved| +|SEMI|reserved|strict-non-reserved|non-reserved| +|SEPARATED|non-reserved|non-reserved|non-reserved| +|SERDE|non-reserved|non-reserved|non-reserved| +|SERDEPROPERTIES|non-reserved|non-reserved|non-reserved| +|SESSION_USER|reserved|non-reserved|reserved| +|SET|non-reserved|non-reserved|reserved| +|SETS|non-reserved|non-reserved|non-reserved| +|SHOW|non-reserved|non-reserved|non-reserved| +|SKEWED|non-reserved|non-reserved|non-reserved| +|SOME|reserved|non-reserved|reserved| +|SORT|non-reserved|non-reserved|non-reserved| +|SORTED|non-reserved|non-reserved|non-reserved| +|START|non-reserved|non-reserved|reserved| +|STATISTICS|non-reserved|non-reserved|non-reserved| +|STORED|non-reserved|non-reserved|non-reserved| +|STRATIFY|non-reserved|non-reserved|non-reserved| +|STRUCT|non-reserved|non-reserved|non-reserved| +|SUBSTR|non-reserved|non-reserved|non-reserved| +|SUBSTRING|non-reserved|non-reserved|non-reserved| +|TABLE|reserved|non-reserved|reserved| +|TABLES|non-reserved|non-reserved|non-reserved| +|TABLESAMPLE|non-reserved|non-reserved|reserved| +|TBLPROPERTIES|non-reserved|non-reserved|non-reserved| +|TEMPORARY|non-reserved|non-reserved|non-reserved| +|TERMINATED|non-reserved|non-reserved|non-reserved| +|THEN|reserved|non-reserved|reserved| +|TO|reserved|non-reserved|reserved| +|TOUCH|non-reserved|non-reserved|non-reserved| +|TRAILING|reserved|non-reserved|reserved| +|TRANSACTION|non-reserved|non-reserved|non-reserved| +|TRANSACTIONS|non-reserved|non-reserved|non-reserved| +|TRANSFORM|non-reserved|non-reserved|non-reserved| +|TRIM|non-reserved|non-reserved|non-reserved| +|TRUE|non-reserved|non-reserved|reserved| +|TRUNCATE|non-reserved|non-reserved|reserved| +|UNARCHIVE|non-reserved|non-reserved|non-reserved| +|UNBOUNDED|non-reserved|non-reserved|non-reserved| +|UNCACHE|non-reserved|non-reserved|non-reserved| +|UNION|reserved|strict-non-reserved|reserved| +|UNIQUE|reserved|non-reserved|reserved| +|UNKNOWN|reserved|non-reserved|reserved| +|UNLOCK|non-reserved|non-reserved|non-reserved| +|UNSET|non-reserved|non-reserved|non-reserved| +|UPDATE|non-reserved|non-reserved|reserved| +|USE|non-reserved|non-reserved|non-reserved| +|USER|reserved|non-reserved|reserved| +|USING|reserved|strict-non-reserved|reserved| +|VALUES|non-reserved|non-reserved|reserved| +|VIEW|non-reserved|non-reserved|non-reserved| +|VIEWS|non-reserved|non-reserved|non-reserved| +|WHEN|reserved|non-reserved|reserved| +|WHERE|reserved|non-reserved|reserved| +|WINDOW|non-reserved|non-reserved|reserved| +|WITH|reserved|non-reserved|reserved| +|YEAR|reserved|non-reserved|reserved| diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md index 0d49f6f88222..3f6b6b590f84 100644 --- a/docs/sql-ref-datatypes.md +++ b/docs/sql-ref-datatypes.md @@ -71,130 +71,24 @@ You can access them by doing {% include_example data_types scala/org/apache/spark/examples/sql/SparkSQLExample.scala %} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Data typeValue type in ScalaAPI to access or create a data type
ByteType Byte - ByteType -
ShortType Short - ShortType -
IntegerType Int - IntegerType -
LongType Long - LongType -
FloatType Float - FloatType -
DoubleType Double - DoubleType -
DecimalType java.math.BigDecimal - DecimalType -
StringType String - StringType -
BinaryType Array[Byte] - BinaryType -
BooleanType Boolean - BooleanType -
TimestampType java.sql.Timestamp - TimestampType -
DateType java.sql.Date - DateType -
ArrayType scala.collection.Seq - ArrayType(elementType, [containsNull])
- Note: The default value of containsNull is true. -
MapType scala.collection.Map - MapType(keyType, valueType, [valueContainsNull])
- Note: The default value of valueContainsNull is true. -
StructType org.apache.spark.sql.Row - StructType(fields)
- Note: fields is a Seq of StructFields. Also, two fields with the same - name are not allowed. -
StructField The value type in Scala of the data type of this field - (For example, Int for a StructField with the data type IntegerType) - StructField(name, dataType, [nullable])
- Note: The default value of nullable is true. -
+|Data type|Value type in Scala|API to access or create a data type| +|---------|-------------------|-----------------------------------| +|**ByteType**|Byte|ByteType| +|**ShortType**|Short|ShortType| +|**IntegerType**|Int|IntegerType| +|**LongType**|Long|LongType| +|**FloatType**|Float|FloatType| +|**DoubleType**|Double|DoubleType| +|**DecimalType**|java.math.BigDecimal|DecimalType| +|**StringType**|String|StringType| +|**BinaryType**|Array[Byte]|BinaryType| +|**BooleanType**|Boolean|BooleanType| +|**TimestampType**|java.sql.Timestamp|TimestampType| +|**DateType**|java.sql.Date|DateType| +|**ArrayType**|scala.collection.Seq|ArrayType(*elementType*, [*containsNull]*)
**Note:** The default value of *containsNull* is true.| +|**MapType**|scala.collection.Map|MapType(*keyType*, *valueType*, [*valueContainsNull]*)
**Note:** The default value of *valueContainsNull* is true.| +|**StructType**|org.apache.spark.sql.Row|StructType(*fields*)
**Note:** *fields* is a Seq of StructFields. Also, two fields with the same name are not allowed.| +|**StructField**|The value type in Scala of the data type of this field(For example, Int for a StructField with the data type IntegerType)|StructField(*name*, *dataType*, [*nullable*])
**Note:** The default value of *nullable* is true.| @@ -205,132 +99,24 @@ All data types of Spark SQL are located in the package of please use factory methods provided in `org.apache.spark.sql.types.DataTypes`. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Data typeValue type in JavaAPI to access or create a data type
ByteType byte or Byte - DataTypes.ByteType -
ShortType short or Short - DataTypes.ShortType -
IntegerType int or Integer - DataTypes.IntegerType -
LongType long or Long - DataTypes.LongType -
FloatType float or Float - DataTypes.FloatType -
DoubleType double or Double - DataTypes.DoubleType -
DecimalType java.math.BigDecimal - DataTypes.createDecimalType()
- DataTypes.createDecimalType(precision, scale). -
StringType String - DataTypes.StringType -
BinaryType byte[] - DataTypes.BinaryType -
BooleanType boolean or Boolean - DataTypes.BooleanType -
TimestampType java.sql.Timestamp - DataTypes.TimestampType -
DateType java.sql.Date - DataTypes.DateType -
ArrayType java.util.List - DataTypes.createArrayType(elementType)
- Note: The value of containsNull will be true
- DataTypes.createArrayType(elementType, containsNull). -
MapType java.util.Map - DataTypes.createMapType(keyType, valueType)
- Note: The value of valueContainsNull will be true.
- DataTypes.createMapType(keyType, valueType, valueContainsNull)
-
StructType org.apache.spark.sql.Row - DataTypes.createStructType(fields)
- Note: fields is a List or an array of StructFields. - Also, two fields with the same name are not allowed. -
StructField The value type in Java of the data type of this field - (For example, int for a StructField with the data type IntegerType) - DataTypes.createStructField(name, dataType, nullable) -
+|Data type|Value type in Java|API to access or create a data type| +|---------|------------------|-----------------------------------| +|**ByteType**|byte or Byte|DataTypes.ByteType| +|**ShortType**|short or Short|DataTypes.ShortType| +|**IntegerType**|int or Integer|DataTypes.IntegerType| +|**LongType**|long or Long|DataTypes.LongType| +|**FloatType**|float or Float|DataTypes.FloatType| +|**DoubleType**|double or Double|DataTypes.DoubleType| +|**DecimalType**|java.math.BigDecimal|DataTypes.createDecimalType()
DataTypes.createDecimalType(*precision*, *scale*).| +|**StringType**|String|DataTypes.StringType| +|**BinaryType**|byte[]|DataTypes.BinaryType| +|**BooleanType**|boolean or Boolean|DataTypes.BooleanType| +|**TimestampType**|java.sql.Timestamp|DataTypes.TimestampType| +|**DateType**|java.sql.Date|DataTypes.DateType| +|**ArrayType**|java.util.List|DataTypes.createArrayType(*elementType*)
**Note:** The value of *containsNull* will be true.
DataTypes.createArrayType(*elementType*, *containsNull*).| +|**MapType**|java.util.Map|DataTypes.createMapType(*keyType*, *valueType*)
**Note:** The value of *valueContainsNull* will be true.
DataTypes.createMapType(*keyType*, *valueType*, *valueContainsNull*)| +|**StructType**|org.apache.spark.sql.Row|DataTypes.createStructType(*fields*)
**Note:** *fields* is a List or an array of StructFields.Also, two fields with the same name are not allowed.| +|**StructField**|The value type in Java of the data type of this field (For example, int for a StructField with the data type IntegerType)|DataTypes.createStructField(*name*, *dataType*, *nullable*)| @@ -342,370 +128,73 @@ You can access them by doing from pyspark.sql.types import * {% endhighlight %} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Data typeValue type in PythonAPI to access or create a data type
ByteType - int or long
- Note: Numbers will be converted to 1-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of -128 to 127. -
- ByteType() -
ShortType - int or long
- Note: Numbers will be converted to 2-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of -32768 to 32767. -
- ShortType() -
IntegerType int or long - IntegerType() -
LongType - long
- Note: Numbers will be converted to 8-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of - -9223372036854775808 to 9223372036854775807. - Otherwise, please convert data to decimal.Decimal and use DecimalType. -
- LongType() -
FloatType - float
- Note: Numbers will be converted to 4-byte single-precision floating - point numbers at runtime. -
- FloatType() -
DoubleType float - DoubleType() -
DecimalType decimal.Decimal - DecimalType() -
StringType string - StringType() -
BinaryType bytearray - BinaryType() -
BooleanType bool - BooleanType() -
TimestampType datetime.datetime - TimestampType() -
DateType datetime.date - DateType() -
ArrayType list, tuple, or array - ArrayType(elementType, [containsNull])
- Note: The default value of containsNull is True. -
MapType dict - MapType(keyType, valueType, [valueContainsNull])
- Note: The default value of valueContainsNull is True. -
StructType list or tuple - StructType(fields)
- Note: fields is a Seq of StructFields. Also, two fields with the same - name are not allowed. -
StructField The value type in Python of the data type of this field - (For example, Int for a StructField with the data type IntegerType) - StructField(name, dataType, [nullable])
- Note: The default value of nullable is True. -
+|Data type|Value type in Python|API to access or create a data type| +|---------|--------------------|-----------------------------------| +|**ByteType**|int or long
**Note:** Numbers will be converted to 1-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -128 to 127.|ByteType()| +|**ShortType**|int or long
**Note:** Numbers will be converted to 2-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -32768 to 32767.|ShortType()| +|**IntegerType**|int or long|IntegerType()| +|**LongType**|long
**Note:** Numbers will be converted to 8-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -9223372036854775808 to 9223372036854775807.Otherwise, please convert data to decimal.Decimal and use DecimalType.|LongType()| +|**FloatType**|float
**Note:** Numbers will be converted to 4-byte single-precision floating point numbers at runtime.|FloatType()| +|**DoubleType**|float|DoubleType()| +|**DecimalType**|decimal.Decimal|DecimalType()| +|**StringType**|string|StringType()| +|**BinaryType**|bytearray|BinaryType()| +|**BooleanType**|bool|BooleanType()| +|**TimestampType**|datetime.datetime|TimestampType()| +|**DateType**|datetime.date|DateType()| +|**ArrayType**|list, tuple, or array|ArrayType(*elementType*, [*containsNull*])
**Note:**The default value of *containsNull* is True.| +|**MapType**|dict|MapType(*keyType*, *valueType*, [*valueContainsNull]*)
**Note:**The default value of *valueContainsNull* is True.| +|**StructType**|list or tuple|StructType(*fields*)
**Note:** *fields* is a Seq of StructFields. Also, two fields with the same name are not allowed.| +|**StructField**|The value type in Python of the data type of this field
(For example, Int for a StructField with the data type IntegerType)|StructField(*name*, *dataType*, [*nullable*])
**Note:** The default value of *nullable* is True.|
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Data typeValue type in RAPI to access or create a data type
ByteType - integer
- Note: Numbers will be converted to 1-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of -128 to 127. -
- "byte" -
ShortType - integer
- Note: Numbers will be converted to 2-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of -32768 to 32767. -
- "short" -
IntegerType integer - "integer" -
LongType - integer
- Note: Numbers will be converted to 8-byte signed integer numbers at runtime. - Please make sure that numbers are within the range of - -9223372036854775808 to 9223372036854775807. - Otherwise, please convert data to decimal.Decimal and use DecimalType. -
- "long" -
FloatType - numeric
- Note: Numbers will be converted to 4-byte single-precision floating - point numbers at runtime. -
- "float" -
DoubleType numeric - "double" -
DecimalType Not supported - Not supported -
StringType character - "string" -
BinaryType raw - "binary" -
BooleanType logical - "bool" -
TimestampType POSIXct - "timestamp" -
DateType Date - "date" -
ArrayType vector or list - list(type="array", elementType=elementType, containsNull=[containsNull])
- Note: The default value of containsNull is TRUE. -
MapType environment - list(type="map", keyType=keyType, valueType=valueType, valueContainsNull=[valueContainsNull])
- Note: The default value of valueContainsNull is TRUE. -
StructType named list - list(type="struct", fields=fields)
- Note: fields is a Seq of StructFields. Also, two fields with the same - name are not allowed. -
StructField The value type in R of the data type of this field - (For example, integer for a StructField with the data type IntegerType) - list(name=name, type=dataType, nullable=[nullable])
- Note: The default value of nullable is TRUE. -
+|Data type|Value type in R|API to access or create a data type| +|---------|---------------|-----------------------------------| +|**ByteType**|integer
**Note:** Numbers will be converted to 1-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -128 to 127.|"byte"| +|**ShortType**|integer
**Note:** Numbers will be converted to 2-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -32768 to 32767.|"short"| +|**IntegerType**|integer|"integer"| +|**LongType**|integer
**Note:** Numbers will be converted to 8-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -9223372036854775808 to 9223372036854775807. Otherwise, please convert data to decimal.Decimal and use DecimalType.|"long"| +|**FloatType**|numeric
**Note:** Numbers will be converted to 4-byte single-precision floating point numbers at runtime.|"float"| +|**DoubleType**|numeric|"double"| +|**DecimalType**|Not supported|Not supported| +|**StringType**|character|"string"| +|**BinaryType**|raw|"binary"| +|**BooleanType**|logical|"bool"| +|**TimestampType**|POSIXct|"timestamp"| +|**DateType**|Date|"date"| +|**ArrayType**|vector or list|list(type="array", elementType=*elementType*, containsNull=[*containsNull*])
**Note:** The default value of *containsNull* is TRUE.| +|**MapType**|environment|list(type="map", keyType=*keyType*, valueType=*valueType*, valueContainsNull=[*valueContainsNull*])
**Note:** The default value of *valueContainsNull* is TRUE.| +|**StructType**|named list|list(type="struct", fields=*fields*)
**Note:** *fields* is a Seq of StructFields. Also, two fields with the same name are not allowed.| +|**StructField**|The value type in R of the data type of this field (For example, integer for a StructField with the data type IntegerType)|list(name=*name*, type=*dataType*, nullable=[*nullable*])
**Note:** The default value of *nullable* is TRUE.| +
The following table shows the type names as well as aliases used in Spark SQL parser for each data type. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Data typeSQL name
BooleanType BOOLEAN
ByteType BYTE, TINYINT
ShortType SHORT, SMALLINT
IntegerType INT, INTEGER
LongType LONG, BIGINT
FloatType FLOAT, REAL
DoubleType DOUBLE
DateType DATE
TimestampType TIMESTAMP
StringType STRING
BinaryType BINARY
DecimalType DECIMAL, DEC, NUMERIC
CalendarIntervalType INTERVAL
ArrayType ARRAY<element_type>
StructType STRUCT<field1_name: field1_type, field2_name: field2_type, ...>
MapType MAP<key_type, value_type>
+|Data type|SQL name| +|---------|--------| +|**BooleanType**|BOOLEAN| +|**ByteType**|BYTE, TINYINT| +|**ShortType**|SHORT, SMALLINT| +|**IntegerType**|INT, INTEGER| +|**LongType**|LONG, BIGINT| +|**FloatType**|FLOAT, REAL| +|**DoubleType**|DOUBLE| +|**DateType**|DATE| +|**TimestampType**|TIMESTAMP| +|**StringType**|STRING| +|**BinaryType**|BINARY| +|**DecimalType**|DECIMAL, DEC, NUMERIC| +|**CalendarIntervalType**|INTERVAL| +|**ArrayType**|ARRAY| +|**StructType**|STRUCT| +|**MapType**|MAP| +
diff --git a/docs/sql-ref-datetime-pattern.md b/docs/sql-ref-datetime-pattern.md index e606383c9432..b65bb1319fb9 100644 --- a/docs/sql-ref-datetime-pattern.md +++ b/docs/sql-ref-datetime-pattern.md @@ -29,7 +29,7 @@ There are several common scenarios for datetime usage in Spark: Spark uses pattern letters in the following table for date and timestamp parsing and formatting: |Symbol|Meaning|Presentation|Examples| -|--- |--- |--- |--- | +|------|-------|------------|--------| |**G**|era|text|AD; Anno Domini; A| |**y**|year|year|2020; 20| |**D**|day-of-year|number|189| @@ -56,10 +56,10 @@ Spark uses pattern letters in the following table for date and timestamp parsing |**X**|zone-offset 'Z' for zero|offset-X|Z; -08; -0830; -08:30; -083015; -08:30:15;| |**x**|zone-offset|offset-x|+0000; -08; -0830; -08:30; -083015; -08:30:15;| |**Z**|zone-offset|offset-Z|+0000; -0800; -08:00;| -|**'**|escape for text|delimiter|| +|**'**|escape for text|delimiter| | |**''**|single quote|literal|'| -|**[**|optional section start||| -|**]**|optional section end||| +|**[**|optional section start| | | +|**]**|optional section end| | | The count of pattern letters determines the format. diff --git a/docs/sql-ref-null-semantics.md b/docs/sql-ref-null-semantics.md index dc48a36cadb3..a93218b14277 100644 --- a/docs/sql-ref-null-semantics.md +++ b/docs/sql-ref-null-semantics.md @@ -49,18 +49,18 @@ other `SQL` constructs. The following illustrates the schema layout and data of a table named `person`. The data contains `NULL` values in -the `age` column and this table will be used in various examples in the sections below. +the `age` column and this table will be used in various examples in the sections below.
**TABLE: person** - - - - - - - - - -
IdNameAge
100Joe30
200MarryNULL
300Mike18
400Fred50
500AlbertNULL
600Michelle30
700Dan50
+ +|Id |Name|Age| +|---|----|---| +|100|Joe|30| +|200|Marry|NULL| +|300|Mike|18| +|400|Fred|50| +|500|Albert|NULL| +|600|Michelle|30| +|700|Dan|50| ### Comparison Operators @@ -71,48 +71,11 @@ equal operator ('<=>'), which returns `False` when one of the operand is `NULL` both the operands are `NULL`. The following table illustrates the behaviour of comparison operators when one or both operands are `NULL`: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Left OperandRight Operand>>==<<=<=>
NULLAny valueNULLNULLNULLNULLNULLFalse
Any valueNULLNULLNULLNULLNULLNULLFalse
NULLNULLNULLNULLNULLNULLNULLTrue
+|Left Operand|Right Operand|> |>= |= |< |<= |<=>| +|------------|--------------|---|---|---|---|---|---| +|NULL|Any value|NULL|NULL|NULL|NULL|NULL|False| +|Any value|NULL|NULL|NULL|NULL|NULL|NULL|False| +|NULL|NULL|NULL|NULL|NULL|NULL|NULL|True| ### Examples @@ -157,55 +120,17 @@ as the arguments and return a `Boolean` value. The following tables illustrate the behavior of logical operators when one or both operands are `NULL`. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Left OperandRight OperandORAND
TrueNULLTrueNULL
FalseNULLNULLFalse
NULLTrueTrueNULL
NULLFalseNULLNULL
NULLNULLNULLNULL
-
- - - - - - - - - -
operandNOT
NULLNULL
+|Left Operand|Right Operand|OR |AND| +|------------|-------------|---|---| +|True|NULL|True|NULL| +|False|NULL|NULL|False| +|NULL|True|True|NULL| +|NULL|False|NULL|NULL| +|NULL|NULL|NULL|NULL| + +|operand|NOT| +|-------|---| +|NULL|NULL| ### Examples @@ -335,7 +260,7 @@ the rules of how `NULL` values are handled by aggregate functions. - `NULL` values are ignored from processing by all the aggregate functions. - Only exception to this rule is COUNT(*) function. - Some aggregate functions return `NULL` when all input values are `NULL` or the input data set - is empty.
The list of these functions is: + is empty.
The list of these functions is: - MAX - MIN - SUM diff --git a/docs/sql-ref-syntax-qry-select-tvf.md b/docs/sql-ref-syntax-qry-select-tvf.md index bbfd870d0cbe..7bad64904593 100644 --- a/docs/sql-ref-syntax-qry-select-tvf.md +++ b/docs/sql-ref-syntax-qry-select-tvf.md @@ -40,7 +40,7 @@ function_name ( expression [ , ... ] ) [ table_alias ]
table_alias
- Specifies a temporary name with an optional column name list.

+ Specifies a temporary name with an optional column name list.

Syntax: [ AS ] table_name [ ( column_name [ , ... ] ) ] @@ -50,31 +50,12 @@ function_name ( expression [ , ... ] ) [ table_alias ] ### Supported Table-valued Functions - - - - - - - - - - - - - - - - - - - - - - - - -
FunctionArgument Type(s)Description
range ( end ) Long Creates a table with a single LongType column named id, containing rows in a range from 0 to end (exclusive) with step value 1.
range ( start, end ) Long, Long Creates a table with a single LongType column named id, containing rows in a range from start to end (exclusive) with step value 1.
range ( start, end, step ) Long, Long, Long Creates a table with a single LongType column named id, containing rows in a range from start to end (exclusive) with step value.
range ( start, end, step, numPartitions ) Long, Long, Long, Int Creates a table with a single LongType column named id, containing rows in a range from start to end (exclusive) with step value, with partition number numPartitions specified.
+|Function|Argument Type(s)|Description| +|--------|----------------|-----------| +|**range** ( *end* )|Long|Creates a table with a single *LongType* column named *id*,
containing rows in a range from 0 to *end* (exclusive) with step value 1.| +|**range** ( *start, end* )|Long, Long|Creates a table with a single *LongType* column named *id*,
containing rows in a range from *start* to *end* (exclusive) with step value 1.| +|**range** ( *start, end, step* )|Long, Long, Long|Creates a table with a single *LongType* column named *id*,
containing rows in a range from *start* to *end* (exclusive) with *step* value.| +|**range** ( *start, end, step, numPartitions* )|Long, Long, Long, Int|Creates a table with a single *LongType* column named *id*,
containing rows in a range from *start* to *end* (exclusive) with *step* value, with partition number *numPartitions* specified.| ### Examples