SQL: Show/desc commands now support table ids (#33363)

Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix #33294

Author: costin

docs/reference/sql/concepts.asciidoc

@@ -9,7 +9,7 @@ NOTE: This documentation while trying to be complete, does assume the reader has
 
 As a general rule, {es-sql} as the name indicates provides a SQL interface to {es}. As such, it follows the SQL terminology and conventions first, whenever possible. However the backing engine itself is {es} for which {es-sql} was purposely created hence why features or concepts that are not available, or cannot be mapped correctly, in SQL appear
 in {es-sql}.
-Last but not least, {es-sql} tries to obey the https://en.wikipedia.org/wiki/Principle_of_least_astonishment[principle of least suprise], though as all things in the world, everything is relative.
+Last but not least, {es-sql} tries to obey the https://en.wikipedia.org/wiki/Principle_of_least_astonishment[principle of least surprise], though as all things in the world, everything is relative.
 
 === Mapping concepts across SQL and {es}
 
@@ -26,7 +26,7 @@ So let's start from the bottom; these roughly are:
 |`column`
 |`field`
 |In both cases, at the lowest level, data is stored in _named_ entries, of a variety of <<sql-data-types, data types>>, containing _one_ value. SQL calls such an entry a _column_ while {es} a _field_.
-Notice that in {es} a field can contain _multiple_ values of the same type (esentially a list) while in SQL, a _column_ can contain _exactly_ one value of said type.
+Notice that in {es} a field can contain _multiple_ values of the same type (essentially a list) while in SQL, a _column_ can contain _exactly_ one value of said type.
 {es-sql} will do its best to preserve the SQL semantic and, depending on the query, reject those that return fields with more than one value.
 
 |`row`
@@ -43,7 +43,7 @@ Notice that in {es} a field can contain _multiple_ values of the same type (esen
 
 |`catalog` or `database`
 |`cluster` instance 
-|In SQL, `catalog` or `database` are used interchangebly and represent a set of schemas that is, a number of tables.
+|In SQL, `catalog` or `database` are used interchangeably and represent a set of schemas that is, a number of tables.
 In {es} the set of indices available are grouped in a `cluster`. The semantics also differ a bit; a `database` is essentially yet another namespace (which can have some implications on the way data is stored) while an {es} `cluster` is a runtime instance, or rather a set of at least one {es} instance (typically running distributed).
 In practice this means that while in SQL one can potentially have multiple catalogs inside an instance, in {es} one is restricted to only _one_.
 
@@ -62,4 +62,4 @@ Multiple clusters, each with its own namespace, connected to each other in a fed
 
 |===
 
-As one can see while the mapping between the concepts are not exactly one to one and the semantics somewhat different, there are more things in common than differences. In fact, thanks to SQL declarative nature, many concepts can move across {es} transparently and the terminology of the two likely to be used interchangebly through-out the rest of the material.
+As one can see while the mapping between the concepts are not exactly one to one and the semantics somewhat different, there are more things in common than differences. In fact, thanks to SQL declarative nature, many concepts can move across {es} transparently and the terminology of the two likely to be used interchangeably through-out the rest of the material.

docs/reference/sql/language/index-patterns.asciidoc

@@ -0,0 +1,70 @@
+[role="xpack"]
+[testenv="basic"]
+[[sql-index-pattern]]
+== Index patterns
+
+{es-sql} supports two types of patterns for matching multiple indices or tables:
+
+* {es} multi-index
+
+The {es} notation for enumerating, including or excluding <<multi-index,multi index syntax>>
+is supported _as long_ as it is quoted or escaped as a table identifier.
+
+For example:
+
+["source","sql",subs="attributes,callouts,macros"]
+----
+include-tagged::{sql-specs}/docs.csv-spec[showTablesEsMultiIndex]
+----
+
+Notice the pattern is surrounded by double quotes `"`. It enumerated `*` meaning all indices however
+it excludes (due to `-`) all indices that start with `l`.
+This notation is very convenient and powerful as it allows both inclusion and exclusion, depending on
+the target naming convention.
+
+* SQL `LIKE` notation
+
+The common `LIKE` statement (including escaping if needed) to match a wildcard pattern, based on one `_`
+or multiple `%` characters.
+
+Using `SHOW TABLES` command again:
+
+["source","sql",subs="attributes,callouts,macros"]
+----
+include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeWildcard]
+----
+
+The pattern matches all tables that start with `emp`. 
+
+This command supports _escaping_ as well, for example:
+
+["source","sql",subs="attributes,callouts,macros"]
+----
+include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeEscape]
+----
+
+Notice how now `emp%` does not match any tables because `%`, which means match zero or more characters,
+has been escaped by `!` and thus becomes an regular char. And since there is no table named `emp%`,
+an empty table is returned.
+
+In a nutshell, the differences between the two type of patterns are:
+
+[cols="^h,^,^",options="header"]
+|===
+| Feature | Multi index | SQL `LIKE`
+
+| Type of quoting    | `"` | `'`
+| Inclusion          | Yes | Yes
+| Exclusion          | Yes | No
+| Enumeration        | Yes | No
+| One char pattern   | No  | `_`
+| Multi char pattern | `*` | `%`
+| Escaping           | No  | `ESCAPE`
+
+|===
+
+Which one to use, is up to you however try to stick to the same one across your queries for consistency.
+
+NOTE: As the query type of quoting between the two patterns is fairly similar (`"` vs `'`), {es-sql} _always_
+requires the keyword `LIKE` for SQL `LIKE` pattern.
+

docs/reference/sql/language/index.asciidoc

@@ -7,6 +7,8 @@ This chapter describes the SQL semantics supported in X-Pack namely:
 
 <<sql-data-types>>:: Data types
 <<sql-commands>>:: Commands
+<<sql-index-patterns>>:: Index patterns
 
 include::data-types.asciidoc[]
 include::syntax/index.asciidoc[]
+include::index-patterns.asciidoc[]

docs/reference/sql/language/syntax/describe-table.asciidoc

@@ -6,14 +6,14 @@
 .Synopsis
 [source, sql]
 ----
-DESCRIBE table
+DESCRIBE [table identifier<1>|[LIKE pattern<2>]]
 ----
 
 or 
 
 [source, sql]
 ----
-DESC table
+DESC [table identifier<1>|[LIKE pattern<2>]]
 ----
 
 

docs/reference/sql/language/syntax/show-columns.asciidoc

@@ -6,9 +6,15 @@
 .Synopsis
 [source, sql]
 ----
-SHOW COLUMNS [ FROM | IN ] ? table 
+SHOW COLUMNS [ FROM | IN ]? [ table identifier<1> | [ LIKE pattern<2> ] ]
 ----
 
+<1> single table identifier or double quoted es multi index
+<2> SQL LIKE pattern
+
+See <<sql-index-pattern, index patterns>> for more information about
+patterns.
+
 .Description
 
 List the columns in table and their data type (and other attributes).
@@ -17,3 +23,4 @@ List the columns in table and their data type (and other attributes).
 ----
 include-tagged::{sql-specs}/docs.csv-spec[showColumns]
 ----
+

docs/reference/sql/language/syntax/show-functions.asciidoc

@@ -6,7 +6,7 @@
 .Synopsis
 [source, sql]
 ----
-SHOW FUNCTIONS [ LIKE? pattern<1>? ]?
+SHOW FUNCTIONS [ LIKE pattern<1>? ]?
 ----
 
 <1> SQL match pattern

docs/reference/sql/language/syntax/show-tables.asciidoc

@@ -6,10 +6,15 @@
 .Synopsis
 [source, sql]
 ----
-SHOW TABLES [ LIKE? pattern<1>? ]?
+SHOW TABLES [ table identifier<1> | [ LIKE pattern<2> ] ]?
 ----
 
-<1> SQL match pattern
+<1> single table identifier or double quoted es multi index
+<2> SQL LIKE pattern
+
+See <<sql-index-pattern, index patterns>> for more information about
+patterns.
+
 
 .Description
 
@@ -20,7 +25,15 @@ List the tables available to the current user and their type.
 include-tagged::{sql-specs}/docs.csv-spec[showTables]
 ----
 
-The `LIKE` clause can be used to restrict the list of names to the given pattern.
+Match multiple indices by using {es} <<multi-index,multi index syntax>>
+notation:
+
+["source","sql",subs="attributes,callouts,macros"]
+----
+include-tagged::{sql-specs}/docs.csv-spec[showTablesEsMultiIndex]
+----
+
+One can also use the `LIKE` clause to restrict the list of names to the given pattern.
 
 The pattern can be an exact match:
 ["source","sql",subs="attributes,callouts,macros"]

x-pack/plugin/sql/src/main/antlr/SqlBase.g4

@@ -50,18 +50,18 @@ statement
         )*
         ')')?
         statement                                                                                         #debug
-    | SHOW TABLES (LIKE? pattern)?                                                                        #showTables
-    | SHOW COLUMNS (FROM | IN) tableIdentifier                                                            #showColumns
-    | (DESCRIBE | DESC) tableIdentifier                                                                   #showColumns
-    | SHOW FUNCTIONS (LIKE? pattern)?                                                                     #showFunctions
+    | SHOW TABLES (tableLike=likePattern | tableIdent=tableIdentifier)?                                   #showTables
+    | SHOW COLUMNS (FROM | IN) (tableLike=likePattern | tableIdent=tableIdentifier)                       #showColumns
+    | (DESCRIBE | DESC) (tableLike=likePattern | tableIdent=tableIdentifier)                              #showColumns
+    | SHOW FUNCTIONS (likePattern)?                                                                       #showFunctions
     | SHOW SCHEMAS                                                                                        #showSchemas
     | SYS CATALOGS                                                                                        #sysCatalogs
-    | SYS TABLES (CATALOG LIKE? clusterPattern=pattern)?
-                 (LIKE? tablePattern=pattern)?
+    | SYS TABLES (CATALOG clusterLike=likePattern)?
+                 (tableLike=likePattern | tableIdent=tableIdentifier)?
                  (TYPE string (',' string)* )?                                                            #sysTables
     | SYS COLUMNS (CATALOG cluster=string)?
-                  (TABLE LIKE? indexPattern=pattern)?
-                  (LIKE? columnPattern=pattern)?                                                          #sysColumns
+                  (TABLE tableLike=likePattern | tableIdent=tableIdentifier)?
+                  (columnPattern=likePattern)?                                                            #sysColumns
     | SYS TYPES                                                                                           #sysTypes
     | SYS TABLE TYPES                                                                                     #sysTableTypes  
     ;
@@ -189,6 +189,10 @@ predicate
     | IS NOT? kind=NULL
     ;
 
+likePattern
+    : LIKE pattern
+    ;
+    
 pattern
     : value=string patternEscape?
     ;

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/analysis/index/IndexResolver.java

@@ -140,49 +140,53 @@ public void resolveNames(String indexWildcard, String javaRegex, EnumSet<IndexTy
         boolean retrieveAliases = CollectionUtils.isEmpty(types) || types.contains(IndexType.ALIAS);
         boolean retrieveIndices = CollectionUtils.isEmpty(types) || types.contains(IndexType.INDEX);
 
+        String[] indices = Strings.commaDelimitedListToStringArray(indexWildcard);
         if (retrieveAliases) {
             GetAliasesRequest aliasRequest = new GetAliasesRequest()
                     .local(true)
-                    .aliases(indexWildcard)
+                    .indices(indices)
+                    .aliases(indices)
                     .indicesOptions(IndicesOptions.lenientExpandOpen());
 
             client.admin().indices().getAliases(aliasRequest, ActionListener.wrap(aliases ->
-                            resolveIndices(indexWildcard, javaRegex, aliases, retrieveIndices, listener),
+                            resolveIndices(indices, javaRegex, aliases, retrieveIndices, listener),
                             ex -> {
                                 // with security, two exception can be thrown:
                                 // INFE - if no alias matches
                                 // security exception is the user cannot access aliases
 
                                 // in both cases, that is allowed and we continue with the indices request
                                 if (ex instanceof IndexNotFoundException || ex instanceof ElasticsearchSecurityException) {
-                                    resolveIndices(indexWildcard, javaRegex, null, retrieveIndices, listener);
+                                    resolveIndices(indices, javaRegex, null, retrieveIndices, listener);
                                 } else {
                                     listener.onFailure(ex);
                                 }
                             }));
         } else {
-            resolveIndices(indexWildcard, javaRegex, null, retrieveIndices, listener);
+            resolveIndices(indices, javaRegex, null, retrieveIndices, listener);
         }
     }
 
-    private void resolveIndices(String indexWildcard, String javaRegex, GetAliasesResponse aliases,
+    private void resolveIndices(String[] indices, String javaRegex, GetAliasesResponse aliases,
             boolean retrieveIndices, ActionListener<Set<IndexInfo>> listener) {
 
         if (retrieveIndices) {
             GetIndexRequest indexRequest = new GetIndexRequest()
                     .local(true)
-                    .indices(indexWildcard)
+                    .indices(indices)
+                    .features(Feature.SETTINGS)
+                    .includeDefaults(false)
                     .indicesOptions(IndicesOptions.lenientExpandOpen());
 
             client.admin().indices().getIndex(indexRequest,
-                    ActionListener.wrap(indices -> filterResults(indexWildcard, javaRegex, aliases, indices, listener),
+                    ActionListener.wrap(response -> filterResults(javaRegex, aliases, response, listener),
                             listener::onFailure));
         } else {
-            filterResults(indexWildcard, javaRegex, aliases, null, listener);
+            filterResults(javaRegex, aliases, null, listener);
         }
     }
 
-    private void filterResults(String indexWildcard, String javaRegex, GetAliasesResponse aliases, GetIndexResponse indices,
+    private void filterResults(String javaRegex, GetAliasesResponse aliases, GetIndexResponse indices,
             ActionListener<Set<IndexInfo>> listener) {
 
         // since the index name does not support ?, filter the results manually
@@ -302,7 +306,6 @@ private static GetIndexRequest createGetIndexRequest(String index) {
         return new GetIndexRequest()
                 .local(true)
                 .indices(Strings.commaDelimitedListToStringArray(index))
-                .features(Feature.MAPPINGS)
                 //lenient because we throw our own errors looking at the response e.g. if something was not resolved
                 //also because this way security doesn't throw authorization exceptions but rather honours ignore_unavailable
                 .indicesOptions(IndicesOptions.lenientExpandOpen());

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/parser/CommandBuilder.java

@@ -121,12 +121,14 @@ public Command visitExplain(ExplainContext ctx) {
 
     @Override
     public Object visitShowFunctions(ShowFunctionsContext ctx) {
-        return new ShowFunctions(source(ctx), visitPattern(ctx.pattern()));
+        return new ShowFunctions(source(ctx), visitLikePattern(ctx.likePattern()));
     }
 
     @Override
     public Object visitShowTables(ShowTablesContext ctx) {
-        return new ShowTables(source(ctx), visitPattern(ctx.pattern()));
+        TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
+        String index = ti != null ? ti.qualifiedIndex() : null;
+        return new ShowTables(source(ctx), index, visitLikePattern(ctx.likePattern()));
     }
 
     @Override
@@ -136,8 +138,9 @@ public Object visitShowSchemas(ShowSchemasContext ctx) {
 
     @Override
     public Object visitShowColumns(ShowColumnsContext ctx) {
-        TableIdentifier identifier = visitTableIdentifier(ctx.tableIdentifier());
-        return new ShowColumns(source(ctx), identifier.index());
+        TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
+        String index = ti != null ? ti.qualifiedIndex() : null;
+        return new ShowColumns(source(ctx), index, visitLikePattern(ctx.likePattern()));
     }
 
     @Override
@@ -172,13 +175,17 @@ else if (value.toUpperCase(Locale.ROOT).equals("TABLE")) {
 
         // if the ODBC enumeration is specified, skip validation
         EnumSet<IndexType> set = types.isEmpty() ? null : EnumSet.copyOf(types);
-        return new SysTables(source(ctx), visitPattern(ctx.clusterPattern), visitPattern(ctx.tablePattern), set, legacyTableType);
+        TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
+        String index = ti != null ? ti.qualifiedIndex() : null;
+        return new SysTables(source(ctx), visitLikePattern(ctx.clusterLike), index, visitLikePattern(ctx.tableLike), set, legacyTableType);
     }
 
     @Override
     public Object visitSysColumns(SysColumnsContext ctx) {
-        Location loc = source(ctx);
-        return new SysColumns(loc, string(ctx.cluster), visitPattern(ctx.indexPattern), visitPattern(ctx.columnPattern));
+        TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
+        String index = ti != null ? ti.qualifiedIndex() : null;
+        return new SysColumns(source(ctx), string(ctx.cluster), index, visitLikePattern(ctx.tableLike),
+                visitLikePattern(ctx.columnPattern));
     }
 
     @Override
@@ -190,4 +197,4 @@ public SysTypes visitSysTypes(SysTypesContext ctx) {
     public Object visitSysTableTypes(SysTableTypesContext ctx) {
         return new SysTableTypes(source(ctx));
     }
-}
+}