[BUG] Flaky IT in ml-commons

[zane-neo]

What is the bug?
Below is a flaky IT which fails time to time, we need to fix this:
https://github.com/opensearch-project/ml-commons/actions/runs/9540553655/job/26292459753?pr=2500#step:6:3006

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[rbhavna]

Below is the exception we are seeing:

Suite: Test class org.opensearch.ml.tools.VisualizationsToolIT
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationNotFound" -Dtests.seed=B77447D33F3C293E -Dtests.security.manager=false -Dtests.locale=is-IS -Dtests.timezone=America/Anguilla -Druntime.java=11
  2> org.opensearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:43309/], URI [/_plugins/_ml/models/29kFJJABf_wnGiLB9K81], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete"}],"type":"status_exception","reason":"Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete"},"status":400}
        at __randomizedtesting.SeedInfo.seed([B77447D33F3C293E:7817E34E380F133E]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:385)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:355)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:330)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:179)

        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:152)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:141)
        at app//org.opensearch.ml.rest.MLCommonsRestTestCase.deleteModel(MLCommonsRestTestCase.java:653)
        at app//org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:75)
        at java.base@11.0.23/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base@11.0.23/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base@11.0.23/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.23/java.lang.reflect.Method.invoke(Method.java:566)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestR

[Hailong-am]

I am able to reproduce with multi nodes test, @b4sjoo did a fix with #2510, but the issue still exists

./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT" -Dtests.security.manager=false -Druntime.java=11 -PnumNodes=5

[ylwu-amzn]

@Hailong-am Thanks, can you help fix ?

[dblock]

[Catch All Triage, attendees 1, 2, 3, 4, 5, 6, 7]

[dhrubo-os]

  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.tools.VisualizationsToolIT_5E95563CAA129985-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1347, maxMBSortInHeap=6.527278970770036, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=nmg, timezone=Europe/Isle_of_Man
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 21.0.5 (64-bit)/cpus=4,threads=1,free=430984096,total=536870912
  2> NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestConnectorToolIT, RestMLCustomModelActionIT, RestMLCustomModelChunkActionIT, RestMLDeleteModelActionIT, RestMLDeleteModelGroupActionIT, RestMLDeleteTaskActionIT, RestMLDeployModelActionIT, RestMLFlowAgentIT, RestMLGetModelActionIT, RestMLGetModelGroupActionIT, RestMLGetTaskActionIT, RestMLGuardrailsIT, RestMLInferenceIngestProcessorIT, RestMLInferenceSearchRequestProcessorIT, RestMLInferenceSearchResponseProcessorIT, RestMLMemoryCircuitBreakerIT, RestMLRAGSearchProcessorIT, RestMLRemoteInferenceIT, RestMLSearchModelActionIT, RestMLSearchTaskActionIT, RestMLTrainAndPredictIT, RestMemoryCreateConversationActionIT, RestMemoryCreateInteractionActionIT, RestMemoryDeleteConversationActionIT, RestMemoryGetConversationActionIT, RestMemoryGetConversationsActionIT, RestMemoryGetInteractionActionIT, RestMemoryGetInteractionsActionIT, RestMemorySearchConversationsActionIT, RestMemorySearchInteractionsActionIT, RestSearchIndexToolIT, VisualizationsToolIT]
  1> [2025-01-09T23:17:35,977][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 1: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:35,977][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 1-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:36,980][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 2: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:36,980][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 2-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:38,983][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 3: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:38,983][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 3-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:41,985][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 4: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:41,985][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 4-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:45,988][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 5: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:45,988][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 5-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:52,206][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] after test
  1> [2025-01-09T23:17:52,218][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] before test
  1> [2025-01-09T23:17:53,740][INFO ][o.o.m.t.ToolIntegrationWithLLMTest] [testVisualizationFound] model_id: n89cTZQBEq7hENXTJ7hJ, agent_id: DSpcTZQBBlnK58luK-_A
  1> [2025-01-09T23:17:54,492][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 1: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:54,492][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 1-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:55,495][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 2: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:55,495][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 2-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:57,498][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 3: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:57,498][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 3-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:00,501][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 4: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:00,501][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 4-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:04,504][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 5: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:04,504][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 5-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:10,744][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] after test


Tests with failures:
 - org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationNotFound
 - org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound
111 tests completed, 2 failed, 10 skipped

This is the log from this workflow: https://github.com/opensearch-project/ml-commons/actions/runs/12699435476/job/35400086583?pr=3307

@Hailong-am could you please take a look into this flaky test? This one started being very frustrating experience. thanks. cc @model-collapse

[Hailong-am]

  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.tools.VisualizationsToolIT_5E95563CAA129985-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1347, maxMBSortInHeap=6.527278970770036, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=nmg, timezone=Europe/Isle_of_Man
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 21.0.5 (64-bit)/cpus=4,threads=1,free=430984096,total=536870912
  2> NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestConnectorToolIT, RestMLCustomModelActionIT, RestMLCustomModelChunkActionIT, RestMLDeleteModelActionIT, RestMLDeleteModelGroupActionIT, RestMLDeleteTaskActionIT, RestMLDeployModelActionIT, RestMLFlowAgentIT, RestMLGetModelActionIT, RestMLGetModelGroupActionIT, RestMLGetTaskActionIT, RestMLGuardrailsIT, RestMLInferenceIngestProcessorIT, RestMLInferenceSearchRequestProcessorIT, RestMLInferenceSearchResponseProcessorIT, RestMLMemoryCircuitBreakerIT, RestMLRAGSearchProcessorIT, RestMLRemoteInferenceIT, RestMLSearchModelActionIT, RestMLSearchTaskActionIT, RestMLTrainAndPredictIT, RestMemoryCreateConversationActionIT, RestMemoryCreateInteractionActionIT, RestMemoryDeleteConversationActionIT, RestMemoryGetConversationActionIT, RestMemoryGetConversationsActionIT, RestMemoryGetInteractionActionIT, RestMemoryGetInteractionsActionIT, RestMemorySearchConversationsActionIT, RestMemorySearchInteractionsActionIT, RestSearchIndexToolIT, VisualizationsToolIT]
  1> [2025-01-09T23:17:35,977][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 1: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:35,977][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 1-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:36,980][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 2: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:36,980][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 2-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:38,983][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 3: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:38,983][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 3-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:41,985][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 4: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:41,985][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 4-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:45,988][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] Attempt 5: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:45,988][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] The 5-th attempt on GET:/_plugins/_ml/models/CipbTZQBBlnK58lu3-9x . response: Response{requestLine=GET /_plugins/_ml/models/CipbTZQBBlnK58lu3-9x HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:52,206][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationNotFound] after test
  1> [2025-01-09T23:17:52,218][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] before test
  1> [2025-01-09T23:17:53,740][INFO ][o.o.m.t.ToolIntegrationWithLLMTest] [testVisualizationFound] model_id: n89cTZQBEq7hENXTJ7hJ, agent_id: DSpcTZQBBlnK58luK-_A
  1> [2025-01-09T23:17:54,492][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 1: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:54,492][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 1-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:40915, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:55,495][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 2: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:55,495][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 2-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36987, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:57,498][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 3: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:17:57,498][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 3-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:43979, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:00,501][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 4: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:00,501][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 4-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://[::1]:36585, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:04,504][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] Attempt 5: Status: OK, Response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:04,504][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] The 5-th attempt on GET:/_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ . response: Response{requestLine=GET /_plugins/_ml/models/n89cTZQBEq7hENXTJ7hJ HTTP/1.1, host=http://127.0.0.1:41449, response=HTTP/1.1 200 OK}
  1> [2025-01-09T23:18:10,744][INFO ][o.o.m.t.VisualizationsToolIT] [testVisualizationFound] after test


Tests with failures:
 - org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationNotFound
 - org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound
111 tests completed, 2 failed, 10 skipped

This is the log from this workflow: https://github.com/opensearch-project/ml-commons/actions/runs/12699435476/job/35400086583?pr=3307

@Hailong-am could you please take a look into this flaky test? This one started being very frustrating experience. thanks. cc @model-collapse

I will take a look and try to fix the test first and trouble shoot model undeploy issue after the test case fix