Skip to content

[ML][Inference] Fix model pagination with models as resources #51573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 31, 2020
Merged

[ML][Inference] Fix model pagination with models as resources #51573

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b8c0d2e
[ML][Inference] Fix model pagination with models as resources
benwtrent Jan 28, 2020
e7e8249
addressing pr comments
benwtrent Jan 29, 2020
5a7052f
addressing PR comments
benwtrent Jan 29, 2020
87f28c9
Simplifying paging logic
benwtrent Jan 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 55 additions & 23 deletions .../src/main/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.client.Client;
import org.elasticsearch.common.CheckedBiFunction;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.collect.Tuple;
Expand Down Expand Up @@ -73,10 +72,10 @@
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;

import static org.elasticsearch.xpack.core.ClientHelper.ML_ORIGIN;
import static org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin;
Expand Down Expand Up @@ -381,19 +380,34 @@ public void deleteTrainedModel(String modelId, ActionListener<Boolean> listener)

public void expandIds(String idExpression,
boolean allowNoResources,
@Nullable PageParams pageParams,
PageParams pageParams,
Copy link
Contributor

@przemekwitek przemekwitek Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, are pageParams guaranteed to be non-null?

Copy link
Member

@davidkyle davidkyle Jan 29, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the pageParams parameter really necessary? I'd argue that I would expect expandIds to return all the ids rather than a page as the id small compared to a full document.

The default value of PageParams.getSize() is 100. Without setting a page param object you may not realise that limitation is in place and that you are not getting all of your ids. Is there even a mechanism to say you need to get the second page now?

Compare to JobConfigProvider.expandJobIds which gives you AnomalyDetectorsIndex.CONFIG_INDEX_MAX_RESULTS_WINDOW = 10_000 ids.

It would also simplify this code to remove to the pageParams which is a very good reason to remove it.

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the pageParams parameter really necessary?

Yes, I think it is. The APIs making calls against this function require paging params (default is from: 0, size: 100). I think this is a sane way of handling larger numbers of items.

Compare to JobConfigProvider.expandJobIds which gives you AnomalyDetectorsIndex.CONFIG_INDEX_MAX_RESULTS_WINDOW = 10_000 ids.

I think that this is a bug and shows that we don't support good pagination at all with anomaly detection jobs.

Is there even a mechanism to say you need to get the second page now?

Yes, the total count of all the ids found with id pattern is returned.

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, are pageParams guaranteed to be non-null?

It is not possible to set them as null via the API since you can only set from and size individually.

If it is null from a client, they just don't set those parameters (and thus get the default value).

If an internal caller is making these calls, they should not explicitly set pageParams to null and if they do, they will receive an error pretty quickly.

Copy link
Member

@davidkyle davidkyle Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern was that with default paging we open ourselves up to bugs such as elastic/kibana#38559.

I assumed getTrainedModel would take paging parameters in which case paging in expandIds would be redundant but I see from the way the code is used that is not the case and GetTrainedModelsStatsAction explicitly depends on paging ids.

AD jobs are not paged because when they lived in clusterstate all where returned by the GET API. It would have been a breaking change to require paging on that API when we move to the .ml-config index and we could not make a breaking change in that release.

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the apis GET _ml/trained_models and GET _ml/trained_models/_stats accept paging parameters and use those when expanding the IDs.

Since they are stored in indices, paging parameters are a must (even for expanding IDs) and might as well have it handled now instead of worrying about adding them later when we reach scaling issues.

Set<String> tags,
ActionListener<Tuple<Long, Set<String>>> idsListener) {
String[] tokens = Strings.tokenizeToStringArray(idExpression, ",");
Set<String> matchedResourceIds = matchedResourceIds(tokens);
Set<String> foundResourceIds;
if (tags.isEmpty()) {
foundResourceIds = matchedResourceIds;
} else {
foundResourceIds = new HashSet<>();
for(String resourceId : matchedResourceIds) {
// Does the model as a resource have all the tags?
if (Sets.newHashSet(loadModelFromResource(resourceId, true).getTags()).containsAll(tags)) {
foundResourceIds.add(resourceId);
}
}
}
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.sort(SortBuilders.fieldSort(TrainedModelConfig.MODEL_ID.getPreferredName())
// If there are no resources, there might be no mapping for the id field.
// This makes sure we don't get an error if that happens.
.unmappedType("long"))
.query(buildExpandIdsQuery(tokens, tags));
if (pageParams != null) {
sourceBuilder.from(pageParams.getFrom()).size(pageParams.getSize());
}
.query(buildExpandIdsQuery(tokens, tags))
// We "buffer" the from and size to take into account models stored as resources.
// This is so we handle the edge cases when the model that is stored as a resource is at the start/end of
// a page.
.from(Math.max(0, pageParams.getFrom() - foundResourceIds.size()))
Copy link
Member

@davidkyle davidkyle Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the strategy is to get extra then figure out where the resource model ids would fit in that sorted list 👍

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidkyle exactly. If we don't have the buffer, AND the resource model ID is at the start of the list, we don't know if it is ACTUALLY at the start of the page, or the END of the previous page.

Hence the need for a buffer.

.size(Math.min(10_000, pageParams.getSize() + foundResourceIds.size()));
sourceBuilder.trackTotalHits(true)
// we only care about the item id's
.fetchSource(TrainedModelConfig.MODEL_ID.getPreferredName(), null);
Expand All @@ -406,47 +420,65 @@ public void expandIds(String idExpression,
indicesOptions.expandWildcardsClosed(),
indicesOptions))
.source(sourceBuilder);
Set<String> foundResourceIds = new LinkedHashSet<>();
if (tags.isEmpty()) {
foundResourceIds.addAll(matchedResourceIds(tokens));
} else {
for(String resourceId : matchedResourceIds(tokens)) {
// Does the model as a resource have all the tags?
if (Sets.newHashSet(loadModelFromResource(resourceId, true).getTags()).containsAll(tags)) {
foundResourceIds.add(resourceId);
}
}
}

executeAsyncWithOrigin(client.threadPool().getThreadContext(),
ML_ORIGIN,
searchRequest,
ActionListener.<SearchResponse>wrap(
response -> {
long totalHitCount = response.getHits().getTotalHits().value + foundResourceIds.size();
Set<String> foundFromDocs = new HashSet<>();
for (SearchHit hit : response.getHits().getHits()) {
Map<String, Object> docSource = hit.getSourceAsMap();
if (docSource == null) {
continue;
}
Object idValue = docSource.get(TrainedModelConfig.MODEL_ID.getPreferredName());
if (idValue instanceof String) {
foundResourceIds.add(idValue.toString());
foundFromDocs.add(idValue.toString());
}
}
Set<String> allFoundIds = collectIds(pageParams, foundResourceIds, foundFromDocs);
ExpandedIdsMatcher requiredMatches = new ExpandedIdsMatcher(tokens, allowNoResources);
requiredMatches.filterMatchedIds(foundResourceIds);
requiredMatches.filterMatchedIds(allFoundIds);
if (requiredMatches.hasUnmatchedIds()) {
idsListener.onFailure(ExceptionsHelper.missingTrainedModel(requiredMatches.unmatchedIdsString()));
Copy link
Member

@davidkyle davidkyle Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this error in a valid situation?

from = 0, size = 10, idExpression = 'bar*,foo'

I have the models bar-1, bar-2 ... bar-10 and foo.

The first 10 bar-n are returned by the search but foo is unmatched.

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidkyle i think this was a problem well before I added this logic. Data frame analytics suffers from this as well.

I think it is OK to error in this case.

I may be wrong though (about it being OK) :D.

Copy link
Member

@davidkyle davidkyle Jan 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly it's not part of this change but let's follow up on it. I don't think we should be reporting en error when we don't know if it is an error or not, maybe we can relax the check

} else {
idsListener.onResponse(Tuple.tuple(totalHitCount, foundResourceIds));

idsListener.onResponse(Tuple.tuple(totalHitCount, allFoundIds));
}
},
idsListener::onFailure
),
client::search);
}

static Set<String> collectIds(PageParams pageParams, Set<String> foundFromResources, Set<String> foundFromDocs) {
// If there are no matching resource models, there was no buffering and the models from the docs
// are paginated correctly.
if (foundFromResources.isEmpty()) {
return foundFromDocs;
}

TreeSet<String> allFoundIds = new TreeSet<>(foundFromDocs);
allFoundIds.addAll(foundFromResources);

if (pageParams.getFrom() > 0) {
// not the first page so there will be extra results at the front to remove
int numToTrimFromFront = Math.min(foundFromResources.size(), pageParams.getFrom());
for (int i = 0; i < numToTrimFromFront; i++) {
allFoundIds.remove(allFoundIds.first());
}
}

// trim down to size removing from the rear
while (allFoundIds.size() > pageParams.getSize()) {
allFoundIds.remove(allFoundIds.last());
}

return allFoundIds;
Copy link
Member

@davidkyle davidkyle Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My way of looking at it is this: we have a request for 10 ids and 2 models stored as resource. We query for 14 ids so we can figure out where the resource model ids would fit. We have a result like

AAOOOOOOOOOOBB

where the As and Bs are the padding.

If the model resource Ids come before the As then they are outside the ordering and all the Os are returned. Same if the model resource ids come after the Bs.

If they are inserted middle (ids M & N) we have an ordering like

AAOOOOOMOOONOOBB

we we take the first 10 after the As.

Same for
AAMOOOOOONOOOOBB
MNAAOOOOOOOOOOBB
AMAOOOOOONOOOOBB
AAOOOOOOOOOOBBMN

I think we have to track the insertion position of the model resource Ids. Collections.binarySearch() would give the insert position.

There is a further complication when the search does not return size hits and you have
AAOOOOOOOOOO
or
AAOOOO

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no BB at the end. moving back from by 2 and increasing size by 2 simply adds two new items to the start because it moves the page view back but it will still include the end of the initially desired page.

If that makes sense.

Copy link
Member Author

@benwtrent benwtrent Jan 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me take your example and run with it. Assuming size is 10.
10 ids and 2 models
Since we decrease from by two, we need to increase the size by two to fill out the page. If we didn't we would end up with
AAOOOOOOOO (2 As, 8 Os)
But, we increase the size to capture those two Os and our search from the docs returns

AAOOOOOOOOOO

If the resource models are inserted middle (ids M & N) we have an ordering like:

AAOOOOOMOOONOO

since our size is 14 in this case, we trim the first two As and then the last two O

Returning to the user

OOOOOMOOON

Here are the results for the rest of your example insertion points

  1. MOOOOOONOO
  2. AAOOOOOOOO
  3. AOOOOOONOO
  4. OOOOOOOOOO

I think 1 and 4 are obvious OK situations.

2 and 3 may seem weird off hand, but they are OK as well, since M or N would have been at the END of the previous page and "pushed" these two buffered IDS onto this page.

I think we have to track the insertion position of the model resource Ids.

I don't think this is possible since we don't know all the page boundries (without the buffers) and knowing the insertion point does not add anything.

Copy link
Member

@davidkyle davidkyle Jan 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. From those examples it seems once you have taken the 2 sorted sets and merged them together there are 2 conditions to consider.

  1. from == 0 in which case we remove from the back of the set until allFoundIds.size <= pageparams.size

In this case there are no As to remove so we trim the set to size from the back

  1. from > 0, remove the number of resource models from the front of the set - this is the As or another id that displaced the As. Then trim the set to size from the back.

With those observations the code could be simplified to

    static Set<String> collectIds(PageParams pageParams, Set<String> foundFromResources, Set<String> foundFromDocs) {
        // If there are no matching resource models, there was no buffering and the models from the docs
        // are paginated correctly.
        if (foundFromResources.isEmpty()) {
            return foundFromDocs;
        }

        TreeSet<String> allFoundIds = new TreeSet<>(foundFromDocs);
        allFoundIds.addAll(foundFromResources);

        if (pageParams.getFrom() > 0) {
            // not the first page so there will be extra results at the front to remove
            int numToTrimFromFront = Math.min(foundFromResources.size(), pageParams.getFrom());
            for (int i = 0; i < numToTrimFromFront; i++) {
                allFoundIds.remove(allFoundIds.first());
            }
        }

        // trim down to size removing from the rear
        while (allFoundIds.size() > pageParams.getSize()) {
            allFoundIds.remove(allFoundIds.last());
        }

        return allFoundIds;
    }

Copy link
Member Author

@benwtrent benwtrent Jan 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidkyle wrote some more test cases, and this solution works great. I modified my code :D

}

static QueryBuilder buildExpandIdsQuery(String[] tokens, Collection<String> tags) {
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery()
.filter(buildQueryIdExpressionQuery(tokens, TrainedModelConfig.MODEL_ID.getPreferredName()));
Expand Down Expand Up @@ -517,7 +549,7 @@ private static QueryBuilder buildQueryIdExpressionQuery(String[] tokens, String

private Set<String> matchedResourceIds(String[] tokens) {
if (Strings.isAllOrWildcard(tokens)) {
return new HashSet<>(MODELS_STORED_AS_RESOURCE);
return MODELS_STORED_AS_RESOURCE;
}

Set<String> matchedModels = new HashSet<>();
Expand All @@ -535,7 +567,7 @@ private Set<String> matchedResourceIds(String[] tokens) {
}
}
}
return matchedModels;
return Collections.unmodifiableSet(matchedModels);
}

private static <T> T handleSearchItem(MultiSearchResponse.Item item,
Expand Down
48 changes: 48 additions & 0 deletions ...test/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelProviderTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,16 @@
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.TermQueryBuilder;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.xpack.core.action.util.PageParams;
import org.elasticsearch.xpack.core.ml.inference.MlInferenceNamedXContentProvider;
import org.elasticsearch.xpack.core.ml.inference.TrainedModelConfig;
import org.elasticsearch.xpack.core.ml.inference.TrainedModelConfigTests;
import org.elasticsearch.xpack.core.ml.job.messages.Messages;

import java.util.Arrays;
import java.util.Collections;
import java.util.HashSet;
import java.util.TreeSet;

import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.instanceOf;
Expand Down Expand Up @@ -86,6 +90,50 @@ public void testExpandIdsQuery() {
});
}

public void testExpandIdsPagination() {
// NOTE: these tests assume that the query pagination results are "buffered"

assertThat(TrainedModelProvider.collectIds(new PageParams(0, 3),
Collections.emptySet(),
new HashSet<>(Arrays.asList("a", "b", "c"))),
equalTo(new TreeSet<>(Arrays.asList("a", "b", "c"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(0, 3),
Collections.singleton("a"),
new HashSet<>(Arrays.asList("b", "c", "d"))),
equalTo(new TreeSet<>(Arrays.asList("a", "b", "c"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(1, 3),
Collections.singleton("a"),
new HashSet<>(Arrays.asList("b", "c", "d"))),
equalTo(new TreeSet<>(Arrays.asList("b", "c", "d"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(1, 1),
Collections.singleton("c"),
new HashSet<>(Arrays.asList("a", "b"))),
equalTo(new TreeSet<>(Arrays.asList("b"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(1, 1),
Collections.singleton("b"),
new HashSet<>(Arrays.asList("a", "c"))),
equalTo(new TreeSet<>(Arrays.asList("b"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(1, 2),
new HashSet<>(Arrays.asList("a", "b")),
new HashSet<>(Arrays.asList("c", "d", "e"))),
equalTo(new TreeSet<>(Arrays.asList("b", "c"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(1, 3),
new HashSet<>(Arrays.asList("a", "b")),
new HashSet<>(Arrays.asList("c", "d", "e"))),
equalTo(new TreeSet<>(Arrays.asList("b", "c", "d"))));

assertThat(TrainedModelProvider.collectIds(new PageParams(2, 3),
new HashSet<>(Arrays.asList("a", "b")),
new HashSet<>(Arrays.asList("c", "d", "e"))),
equalTo(new TreeSet<>(Arrays.asList("c", "d", "e"))));
}

public void testGetModelThatExistsAsResourceButIsMissing() {
TrainedModelProvider trainedModelProvider = new TrainedModelProvider(mock(Client.class), xContentRegistry());
ElasticsearchException ex = expectThrows(ElasticsearchException.class,
Expand Down
132 changes: 129 additions & 3 deletions x-pack/plugin/src/test/resources/rest-api-spec/test/ml/inference_crud.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,56 @@ setup:
}
}
}

- do:
headers:
Authorization: "Basic eF9wYWNrX3Jlc3RfdXNlcjp4LXBhY2stdGVzdC1wYXNzd29yZA==" # run as x_pack_rest_user, i.e. the test setup superuser
ml.put_trained_model:
model_id: yyy-classification-model
body: >
{
"description": "empty model for tests",
"input": {"field_names": ["field1", "field2"]},
"tags": ["classification", "tag3"],
"definition": {
"preprocessors": [],
"trained_model": {
"tree": {
"feature_names": ["field1", "field2"],
"tree_structure": [
{"node_index": 0, "leaf_value": 1}
],
"target_type": "classification",
"classification_labels": ["no", "yes"]
}
}
}
}

- do:
headers:
Authorization: "Basic eF9wYWNrX3Jlc3RfdXNlcjp4LXBhY2stdGVzdC1wYXNzd29yZA==" # run as x_pack_rest_user, i.e. the test setup superuser
ml.put_trained_model:
model_id: zzz-classification-model
body: >
{
"description": "empty model for tests",
"input": {"field_names": ["field1", "field2"]},
"tags": ["classification", "tag3"],
"definition": {
"preprocessors": [],
"trained_model": {
"tree": {
"feature_names": ["field1", "field2"],
"tree_structure": [
{"node_index": 0, "leaf_value": 1}
],
"target_type": "classification",
"classification_labels": ["no", "yes"]
}
}
}
}
---
"Test get given missing trained model":

Expand Down Expand Up @@ -102,15 +152,20 @@ setup:
- do:
ml.get_trained_models:
model_id: "*"
- match: { count: 4 }
- match: { count: 6 }
- length: { trained_model_configs: 6 }
- match: { trained_model_configs.0.model_id: "a-classification-model" }
- match: { trained_model_configs.1.model_id: "a-regression-model-0" }
- match: { trained_model_configs.2.model_id: "a-regression-model-1" }
- match: { trained_model_configs.3.model_id: "lang_ident_model_1" }
- match: { trained_model_configs.4.model_id: "yyy-classification-model" }
- match: { trained_model_configs.5.model_id: "zzz-classification-model" }

- do:
ml.get_trained_models:
model_id: "a-regression*"
- match: { count: 2 }
- length: { trained_model_configs: 2 }
- match: { trained_model_configs.0.model_id: "a-regression-model-0" }
- match: { trained_model_configs.1.model_id: "a-regression-model-1" }

Expand All @@ -119,7 +174,8 @@ setup:
model_id: "*"
from: 0
size: 2
- match: { count: 4 }
- match: { count: 6 }
- length: { trained_model_configs: 2 }
- match: { trained_model_configs.0.model_id: "a-classification-model" }
- match: { trained_model_configs.1.model_id: "a-regression-model-0" }

Expand All @@ -128,8 +184,78 @@ setup:
model_id: "*"
from: 1
size: 1
- match: { count: 4 }
- match: { count: 6 }
- length: { trained_model_configs: 1 }
- match: { trained_model_configs.0.model_id: "a-regression-model-0" }

- do:
ml.get_trained_models:
model_id: "*"
from: 2
size: 2
- match: { count: 6 }
- length: { trained_model_configs: 2 }
- match: { trained_model_configs.0.model_id: "a-regression-model-1" }
- match: { trained_model_configs.1.model_id: "lang_ident_model_1" }

- do:
ml.get_trained_models:
model_id: "*"
from: 3
size: 1
- match: { count: 6 }
- length: { trained_model_configs: 1 }
- match: { trained_model_configs.0.model_id: "lang_ident_model_1" }

- do:
ml.get_trained_models:
model_id: "*"
from: 3
size: 2
- match: { count: 6 }
- length: { trained_model_configs: 2 }
- match: { trained_model_configs.0.model_id: "lang_ident_model_1" }
- match: { trained_model_configs.1.model_id: "yyy-classification-model" }

- do:
ml.get_trained_models:
model_id: "*"
from: 4
size: 2
- match: { count: 6 }
- length: { trained_model_configs: 2 }
- match: { trained_model_configs.0.model_id: "yyy-classification-model" }
- match: { trained_model_configs.1.model_id: "zzz-classification-model" }

- do:
ml.get_trained_models:
model_id: "a-*,lang*,zzz*"
allow_no_match: true
from: 3
size: 1
- match: { count: 5 }
- length: { trained_model_configs: 1 }
- match: { trained_model_configs.0.model_id: "lang_ident_model_1" }

- do:
ml.get_trained_models:
model_id: "a-*,lang*,zzz*"
allow_no_match: true
from: 4
size: 1
- match: { count: 5 }
- length: { trained_model_configs: 1 }
- match: { trained_model_configs.0.model_id: "zzz-classification-model" }

- do:
ml.get_trained_models:
model_id: "a-*,lang*,zzz*"
from: 4
size: 100
- match: { count: 5 }
- length: { trained_model_configs: 1 }
- match: { trained_model_configs.0.model_id: "zzz-classification-model" }

---
"Test get models with tags":
- do:
Expand Down