Skip to content

Commit b63b50b

Browse files
jpountzjpountz
authored
Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. (#37871)
Currently if you mix typed templates and typeless index creation or typeless templates and typed index creation then you will end up with an error because Elasticsearch tries to create an index that has multiple types: `_doc` and the explicit type name that you used. This commit proposes to give precedence to the index creation call so that the type from the template will be ignored if the index creation call is typeless while the template is typed, and the type from the index creation call will be used if there is a typeless template. This is consistent with the fact that index creation already "wins" if a field is defined differently in the index creation call and in a template: the definition from the index creation call is used in such cases. Closes #37773
1 parent 4dee3f7 commit b63b50b

File tree

4 files changed

+259
-0
lines changed

4 files changed

+259
-0
lines changed

‎docs/reference/mapping/removal_of_types.asciidoc‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -534,3 +534,77 @@ PUT index/_doc/1
534534
The <<docs-index_,GET>>, <<docs-delete,`DELETE`>>, <<docs-update,`_update`>> and <<search,`_search`>> APIs
535535
will continue to return a `_type` key in the response in 7.0, but it is considered deprecated and will be
536536
removed in 8.0.
537+
538+
[float]
539+
=== Index templates
540+
541+
It is recommended to make index templates typeless before upgrading to 7.0 by
542+
re-adding them with `include_type_name` set to `false`.
543+
544+
In case typeless templates are used with typed index creation calls or typed
545+
templates are used with typeless index creation calls, the template will still
546+
be applied but the index creation call decides whether there should be a type
547+
or not. For instance in the below example, `index-1-01` will have a type in
548+
spite of the fact that it matches a template that is typeless, and `index-2-01`
549+
will be typeless in spite of the fact that it matches a template that defines
550+
a type. Both `index-1-01` and `index-2-01` will inherit the `foo` field from
551+
the template that they match.
552+
553+
[source,js]
554+
--------------------------------------------------
555+
PUT _template/template1
556+
{
557+
"index_patterns":[ "index-1-*" ],
558+
"mappings": {
559+
"properties": {
560+
"foo": {
561+
"type": "keyword"
562+
}
563+
}
564+
}
565+
}
566+
567+
PUT _template/template2?include_type_name=true
568+
{
569+
"index_patterns":[ "index-2-*" ],
570+
"mappings": {
571+
"type": {
572+
"properties": {
573+
"foo": {
574+
"type": "keyword"
575+
}
576+
}
577+
}
578+
}
579+
}
580+
581+
PUT index-1-01?include_type_name=true
582+
{
583+
"mappings": {
584+
"type": {
585+
"properties": {
586+
"bar": {
587+
"type": "long"
588+
}
589+
}
590+
}
591+
}
592+
}
593+
594+
PUT index-2-01
595+
{
596+
"mappings": {
597+
"properties": {
598+
"bar": {
599+
"type": "long"
600+
}
601+
}
602+
}
603+
}
604+
--------------------------------------------------
605+
// CONSOLE
606+
607+
In case of implicit index creation, because of documents that get indexed in
608+
an index that doesn't exist yet, the template is always honored. This is
609+
usually not a problem due to the fact that typless index calls work on typed
610+
indices.

‎rest-api-spec/src/main/resources/rest-api-spec/test/indices.create/20_mix_typeless_typeful.yml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
---
2+
"Create a typeless index while there is a typed template":
3+
4+
- skip:
5+
version: " - 6.99.99"
6+
reason: needs change to be backported to 6.7
7+
8+
- do:
9+
indices.put_template:
10+
include_type_name: true
11+
name: test_template
12+
body:
13+
index_patterns: test-*
14+
mappings:
15+
my_type:
16+
properties:
17+
foo:
18+
type: keyword
19+
20+
- do:
21+
indices.create:
22+
include_type_name: false
23+
index: test-1
24+
body:
25+
mappings:
26+
properties:
27+
bar:
28+
type: "long"
29+
30+
- do:
31+
indices.get_mapping:
32+
include_type_name: true
33+
index: test-1
34+
35+
- is_true: test-1.mappings._doc # the index creation call won
36+
- is_false: test-1.mappings.my_type
37+
- is_true: test-1.mappings._doc.properties.foo
38+
- is_true: test-1.mappings._doc.properties.bar
39+
40+
---
41+
"Create a typed index while there is a typeless template":
42+
43+
- skip:
44+
version: " - 6.99.99"
45+
reason: needs change to be backported to 6.7
46+
47+
- do:
48+
indices.put_template:
49+
include_type_name: false
50+
name: test_template
51+
body:
52+
index_patterns: test-*
53+
mappings:
54+
properties:
55+
foo:
56+
type: keyword
57+
58+
- do:
59+
indices.create:
60+
include_type_name: true
61+
index: test-1
62+
body:
63+
mappings:
64+
my_type:
65+
properties:
66+
bar:
67+
type: "long"
68+
69+
- do:
70+
indices.get_mapping:
71+
include_type_name: true
72+
index: test-1
73+
74+
- is_true: test-1.mappings.my_type # the index creation call won
75+
- is_false: test-1.mappings._doc
76+
- is_true: test-1.mappings.my_type.properties.foo
77+
- is_true: test-1.mappings.my_type.properties.bar
78+
79+
---
80+
"Implicitly create a typed index while there is a typeless template":
81+
82+
- skip:
83+
version: " - 6.99.99"
84+
reason: needs change to be backported to 6.7
85+
86+
- do:
87+
indices.put_template:
88+
include_type_name: false
89+
name: test_template
90+
body:
91+
index_patterns: test-*
92+
mappings:
93+
properties:
94+
foo:
95+
type: keyword
96+
97+
- do:
98+
catch: /the final mapping would have more than 1 type/
99+
index:
100+
index: test-1
101+
type: my_type
102+
body: { bar: 42 }
103+
104+
---
105+
"Implicitly create a typeless index while there is a typed template":
106+
107+
- skip:
108+
version: " - 6.99.99"
109+
reason: needs typeless index operations to work on typed indices
110+
111+
- do:
112+
indices.put_template:
113+
include_type_name: true
114+
name: test_template
115+
body:
116+
index_patterns: test-*
117+
mappings:
118+
my_type:
119+
properties:
120+
foo:
121+
type: keyword
122+
123+
- do:
124+
index:
125+
index: test-1
126+
type: my_type
127+
body: { bar: 42 }
128+
129+
- do:
130+
indices.get_mapping:
131+
include_type_name: true
132+
index: test-1
133+
134+
- is_true: test-1.mappings.my_type # the template is honored
135+
- is_false: test-1.mappings._doc
136+
- is_true: test-1.mappings.my_type.properties.foo
137+
- is_true: test-1.mappings.my_type.properties.bar

‎server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,28 @@ public ClusterState execute(ClusterState currentState) throws Exception {
318318
if (mappings.containsKey(cursor.key)) {
319319
XContentHelper.mergeDefaults(mappings.get(cursor.key),
320320
MapperService.parseMapping(xContentRegistry, mappingString));
321+
} else if (mappings.size() == 1 && cursor.key.equals(MapperService.SINGLE_MAPPING_NAME)) {
322+
// Typeless template with typed mapping
323+
Map<String, Object> templateMapping = MapperService.parseMapping(xContentRegistry, mappingString);
324+
assert templateMapping.size() == 1 : templateMapping;
325+
assert cursor.key.equals(templateMapping.keySet().iterator().next()) :
326+
cursor.key + " != " + templateMapping;
327+
Map.Entry<String, Map<String, Object>> mappingEntry = mappings.entrySet().iterator().next();
328+
templateMapping = Collections.singletonMap(
329+
mappingEntry.getKey(), // reuse type name from the mapping
330+
templateMapping.values().iterator().next()); // but actual mappings from the template
331+
XContentHelper.mergeDefaults(mappingEntry.getValue(), templateMapping);
332+
} else if (template.mappings().size() == 1 && mappings.containsKey(MapperService.SINGLE_MAPPING_NAME)) {
333+
// Typed template with typeless mapping
334+
Map<String, Object> templateMapping = MapperService.parseMapping(xContentRegistry, mappingString);
335+
assert templateMapping.size() == 1 : templateMapping;
336+
assert cursor.key.equals(templateMapping.keySet().iterator().next()) :
337+
cursor.key + " != " + templateMapping;
338+
Map<String, Object> mapping = mappings.get(MapperService.SINGLE_MAPPING_NAME);
339+
templateMapping = Collections.singletonMap(
340+
MapperService.SINGLE_MAPPING_NAME, // make template mapping typeless
341+
templateMapping.values().iterator().next());
342+
XContentHelper.mergeDefaults(mapping, templateMapping);
321343
} else {
322344
mappings.put(cursor.key,
323345
MapperService.parseMapping(xContentRegistry, mappingString));

‎server/src/test/java/org/elasticsearch/cluster/metadata/IndexCreationTaskTests.java‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,32 @@ public void testWriteIndexValidationException() throws Exception {
310310
assertThat(exception.getMessage(), startsWith("alias [alias1] has more than one write index ["));
311311
}
312312

313+
public void testTypelessTemplateWithTypedIndexCreation() throws Exception {
314+
addMatchingTemplate(builder -> builder.putMapping("type", "{\"type\": {}}"));
315+
setupRequestMapping(MapperService.SINGLE_MAPPING_NAME, new CompressedXContent("{\"_doc\":{}}"));
316+
executeTask();
317+
assertThat(getMappingsFromResponse(), Matchers.hasKey(MapperService.SINGLE_MAPPING_NAME));
318+
}
319+
320+
public void testTypedTemplateWithTypelessIndexCreation() throws Exception {
321+
addMatchingTemplate(builder -> builder.putMapping(MapperService.SINGLE_MAPPING_NAME, "{\"_doc\": {}}"));
322+
setupRequestMapping("type", new CompressedXContent("{\"type\":{}}"));
323+
executeTask();
324+
assertThat(getMappingsFromResponse(), Matchers.hasKey("type"));
325+
}
326+
327+
public void testTypedTemplate() throws Exception {
328+
addMatchingTemplate(builder -> builder.putMapping("type", "{\"type\": {}}"));
329+
executeTask();
330+
assertThat(getMappingsFromResponse(), Matchers.hasKey("type"));
331+
}
332+
333+
public void testTypelessTemplate() throws Exception {
334+
addMatchingTemplate(builder -> builder.putMapping(MapperService.SINGLE_MAPPING_NAME, "{\"_doc\": {}}"));
335+
executeTask();
336+
assertThat(getMappingsFromResponse(), Matchers.hasKey(MapperService.SINGLE_MAPPING_NAME));
337+
}
338+
313339
private IndexRoutingTable createIndexRoutingTableWithStartedShards(Index index) {
314340
final IndexRoutingTable idxRoutingTable = mock(IndexRoutingTable.class);
315341

0 commit comments

Comments
 (0)