[CI] DocsClientYamlTestSuiteIT test {yaml=reference/rest-api/usage/line_38} failing due to invalid script in pipeline

[cbuescher]

I cannot reproduce this on the commit on master locally, maybe a one-of error?
This comes from docs calling "xpack/_usage" but from the stack trace its the TransportGetTrainedModelsStatsAction that is calling using the failing script, so tagging ML for inspection

Build scan:
https://gradle-enterprise.elastic.co/s/q5ypvd5xlnuey/tests/:docs:integTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Frest-api%2Fusage%2Fline_38%7D

Reproduction line:
./gradlew ':docs:integTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/rest-api/usage/line_38}" -Dtests.seed=6A098B0603E56D9F -Dtests.locale=ga-IE -Dtests.timezone=America/Kralendijk -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Dreference/rest-api/usage/line_38%7D

Failure excerpt:

java.lang.AssertionError: Failure at [reference/rest-api/usage:10]: expected [2xx] status code but api [raw[method=GET path=_xpack/usage]] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"script_exception","reason":"compile error","processor_type":"drop","script_stack":["\n      TimestampHour date =  doc['@timestamp'] ...","                     ^---- HERE"],"script":"\n      TimestampHour date =  doc['@timestamp'].value;\n      return date.getHour()\n    ","lang":"painless","position":{"offset":21,"start":0,"end":46},"stack_trace":"org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"}],"type":"exception","reason":"unexpected failure gathering pipeline information","caused_by":{"type":"script_exception","reason":"compile error","processor_type":"drop","script_stack":["\n      TimestampHour date =  doc['@timestamp'] ...","                     ^---- HERE"],"script":"\n      TimestampHour date =  doc['@timestamp'].value;\n      return date.getHour()\n    ","lang":"painless","position":{"offset":21,"start":0,"end":46},"caused_by":{"type":"illegal_argument_exception","reason":"invalid declaration: cannot resolve type [TimestampHour]","stack_trace":"java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"},"stack_trace":"org.elasticsearch.ElasticsearchException: unexpected failure gathering pipeline information\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:287)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\t... 17 more\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"},"status":500}]

  at __randomizedtesting.SeedInfo.seed([6A098B0603E56D9F:E25DB4DCAD190067]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:489)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
  at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(null:-1)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=GET path=_xpack/usage]] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"script_exception","reason":"compile error","processor_type":"drop","script_stack":["\n      TimestampHour date =  doc['@timestamp'] ...","                     ^---- HERE"],"script":"\n      TimestampHour date =  doc['@timestamp'].value;\n      return date.getHour()\n    ","lang":"painless","position":{"offset":21,"start":0,"end":46},"stack_trace":"org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"}],"type":"exception","reason":"unexpected failure gathering pipeline information","caused_by":{"type":"script_exception","reason":"compile error","processor_type":"drop","script_stack":["\n      TimestampHour date =  doc['@timestamp'] ...","                     ^---- HERE"],"script":"\n      TimestampHour date =  doc['@timestamp'].value;\n      return date.getHour()\n    ","lang":"painless","position":{"offset":21,"start":0,"end":46},"caused_by":{"type":"illegal_argument_exception","reason":"invalid declaration: cannot resolve type [TimestampHour]","stack_trace":"java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"},"stack_trace":"org.elasticsearch.ElasticsearchException: unexpected failure gathering pipeline information\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:287)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1421)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:181)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.newResponse(TransportNodesAction.java:156)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.lambda$finishHim$0(TransportNodesAction.java:291)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.script.ScriptException: compile error\n\tat org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)\n\tat org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)\n\tat org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)\n\tat org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)\n\tat org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)\n\tat org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)\n\t... 17 more\nCaused by: java.lang.IllegalArgumentException: invalid declaration: cannot resolve type [TimestampHour]\n\tat org.elasticsearch.painless.PainlessScript$Script.compile(my-stored-script:22)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:723)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclaration(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclaration.visit(SDeclaration.java:47)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:700)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitDeclBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SDeclBlock.visit(SDeclBlock.java:36)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:327)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitBlock(DefaultSemanticAnalysisPhase.java:158)\n\tat org.elasticsearch.painless.node.SBlock.visit(SBlock.java:37)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visit(DefaultSemanticAnalysisPhase.java:182)\n\tat org.elasticsearch.painless.phase.PainlessSemanticAnalysisPhase.visitFunction(PainlessSemanticAnalysisPhase.java:92)\n\tat org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitClass(DefaultSemanticAnalysisPhase.java:227)\n\tat org.elasticsearch.painless.Compiler.compile(Compiler.java:215)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:401)\n\tat org.elasticsearch.painless.PainlessScriptEngine$2.run(PainlessScriptEngine.java:397)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:399)\n\tat org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:397)\n\t... 29 more\n"},"status":500}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:374)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:478)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
    at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(null:-1)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[cbuescher]

The automatic Failure excerpt is not very readable here, this is the stack with the failing script:

java.lang.AssertionError: Failure at [reference/rest-api/usage:10]: expected [2xx] status code but api [raw[method=GET path=_xpack/usage]] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"script_exception","reason":"compile error","processor_type":"drop","script_stack":["\n      TimestampHour date =  doc['@timestamp'] ...","                     ^---- HERE"],"script":"\n      TimestampHour date =  doc['@timestamp'].value;\n      return date.getHour()\n    ","lang":"painless","position":{"offset":21,"start":0,"end":46},"stack_trace":"org.elasticsearch.script.ScriptException: compile error
org.elasticsearch.painless.PainlessScriptEngine.convertToScriptException(PainlessScriptEngine.java:495)
org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:406)
org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:129)
org.elasticsearch.script.ScriptCache.lambda$compile$0(ScriptCache.java:108)
org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
org.elasticsearch.script.ScriptCache.compile(ScriptCache.java:90)
org.elasticsearch.script.ScriptService.compile(ScriptService.java:636)
org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:77)
org.elasticsearch.ingest.ConditionalProcessor.<init>(ConditionalProcessor.java:58)
org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:595)
org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:547)
org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:467)
org.elasticsearch.ingest.Pipeline.create(Pipeline.java:82)
org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$pipelineIdsByModelIdsOrAliases$16(TransportGetTrainedModelsStatsAction.java:272)
java.base/java.util.HashMap.forEach(HashMap.java:1421)
java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)
org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.pipelineIdsByModelIdsOrAliases(TransportGetTrainedModelsStatsAction.java:270)
org.elasticsearch.xpack.ml.action.TransportGetTrainedModelsStatsAction.lambda$doExecute$4(TransportGetTrainedModelsStatsAction.java:133)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
org.elasticsearch.client.internal.node.NodeClient.lambda$executeLocally$0(NodeClient.java:107)
org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:176)
org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)
org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447)

[benwtrent]

This is not ML related. Something created a pipeline with a drop processor that has the script TimestampHour date = doc['@timestamp']

ML only shows it because we build the pipelines that are stored to extract specific information around ML related processors. We wrap the pipeline building in try{}catch{}, but since this is an AssertionError it bubbles out in tests.

I searched the code base and found:

elasticsearch/docs/reference/scripting/apis/create-stored-script-api.asciidoc

Line 17 in 3d1cb49

TimestampHour date = doc['@timestamp'].value;

So, there is a scripting doc that references this invalid script it seems.

But, I am not sure who is creating that drop processor and which pipeline it is. It may be some earlier docs pipeline or one created automatically by some other plugin.

[droberts195]

The drop processor is in

elasticsearch/docs/reference/ingest.asciidoc

Lines 809 to 837 in 3d1cb49

	PUT _scripts/my-stored-script
	{
	"script": {
	"lang": "painless",
	"source": """
	Collection tags = ctx.tags;
	if(tags != null){
	for (String tag : tags) {
	if (tag.toLowerCase().contains('prod')) {
	return false;
	}
	}
	}
	return true;
	"""
	}
	}
	PUT _ingest/pipeline/my-pipeline
	{
	"processors": [
	{
	"drop": {
	"description": "If 'url.scheme' is 'http', set 'url.insecure' to true",
	"if": { "id": "my-stored-script" }
	}
	}
	]
	}

That also defines my-stored-script, but differently to

elasticsearch/docs/reference/scripting/apis/create-stored-script-api.asciidoc

Lines 12 to 21 in 3d1cb49

	PUT _scripts/my-stored-script
	{
	"script": {
	"lang": "painless",
	"source": """
	TimestampHour date = doc['@timestamp'].value;
	return date.getHour()
	"""
	}
	}

So I think there are two things here:

1. The script in create-stored-script-api.asciidoc needs fixing so it's valid.
2. It seems like the docs test framework cannot cope with the same stored script name being defined in different files, presumably because it doesn't delete stored scripts between tests. So one of the two scripts should be renamed.

@jrodewig it looks like you refactored these docs most recently, albeit 9 months ago. I'm not sure why this problem didn't come up during that 9 months but please could you have a look.

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[cbuescher]

Going back a bit in the test failure history, I see the compile failure already a few times in the last month

[jrodewig]

Thanks for the ping and apologies for the noise. I'll fix up the invalid script and rename the ingest script to avoid naming collisions.