CorruptedFileIT.testCorruptFileThenSnapshotAndRestore fails occasionally

[danielmitterdorfer]

CorruptedFileIT.testCorruptFileThenSnapshotAndRestore fails occasionally with the following assertion error:

   > Throwable #1: java.lang.AssertionError: [test][0], node[kXBrV8JKTCG8ZrjOt_W0vg], [P], s[STARTED], a[id=Zz-bwJLZTkG3yl9D8jMQSw]
   > Expected: <1>
   >      but: was <0>
   >    at __randomizedtesting.SeedInfo.seed([A0A4AF57F24AD88E:6D7015D6605692ED]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.index.store.CorruptedFileIT.listShardFiles(CorruptedFileIT.java:692)
   >    at org.elasticsearch.index.store.CorruptedFileIT.testCorruptFileThenSnapshotAndRestore(CorruptedFileIT.java:510)
   >    at java.lang.Thread.run(Thread.java:745)

This means that the node stats for the cluster node cannot be retrieved (or rather that there are no nodes in the corresponding node stats instance).

Latest build where this occurred: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+seq_no+periodic/1606/consoleFull

[imotov]

This test failure seems to be caused by an exception in FsService which is getting swallowed. I changed the log levels for this test to see which exception is actually getting logged.

[danielmitterdorfer]

Thanks for digging @imotov.

[imotov]

It failed again and @pickypg and I dug a bit more into it and at the moment the only reasonable explanation that we could come up with is some problem in exception handling of stats operation. @pickypg is going to continue digging.

[javanna]

given that this issue had no activity in ~10 months, shall we close it @pickypg @imotov ?

[pickypg]

With #25017 we should catch this if it happens again, so yeah.