ShrinkIndexIT.testCreateShrinkIndexToN fails on Windows

[alpar-t]

5.6

FAILURE 33.4s J0 | ShrinkIndexIT.testCreateShrinkIndexToN <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: timed out waiting for green state
   > 	at __randomizedtesting.SeedInfo.seed([FAAB0749B4C53E6F:EAEB4985116575FC]:0)
   > 	at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:933)
   > 	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:899)
   > 	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:888)
   > 	at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testCreateShrinkIndexToN(ShrinkIndexIT.java:115)
   > 	at java.lang.Thread.run(Thread.java:748)

and

 Caused by: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\recovery.AWXo49VofFhOtlDMUmb5._0.cfs -> C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\_0.cfs
  1> 	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) ~[?:?]
  1> 	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?]
  1> 	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) ~[?:?]
  1> 	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) ~[?:?]
  1> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:147) ~[lucene-test-framework-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:40]
  1> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:147) ~[lucene-test-framework-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:40]
  1> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:147) ~[lucene-test-framework-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:40]
  1> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:147) ~[lucene-test-framework-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:40]
  1> 	at java.nio.file.Files.move(Files.java:1395) ~[?:1.8.0_181]
  1> 	at org.apache.lucene.store.FSDirectory.rename(FSDirectory.java:297) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
  1> 	at org.apache.lucene.store.FilterDirectory.rename(FilterDirectory.java:88) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
  1> 	at org.apache.lucene.store.FilterDirectory.rename(FilterDirectory.java:88) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
  1> 	at org.elasticsearch.index.store.Store.renameTempFilesSafe(Store.java:319) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.RecoveryTarget.renameAllTempFiles(RecoveryTarget.java:181) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:406) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:486) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:480) ~[main/:?]
  1> 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[main/:?]
  1> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[main/:?]
  1> 	at org.elasticsearch.transport.local.LocalTransport$2.doRun(LocalTransport.java:390) ~[main/:?]
  1> 	... 5 more

[...]

1> java.io.IOException: Could not remove the following files (in the order of attempts):
  1>    C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\_0.cfs
  1>    C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\_1.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index\_1.cfs
  1>    C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0\index
  2> REPRODUCE WITH: gradlew :core:integTest -Dtests.seed=FAAB0749B4C53E6F -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT -Dtests.method="testCreateShrinkIndexToN" -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT+10
  1>    C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg\0
  1>    C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+5.6+multijob-windows-compatibility\core\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_FAAB0749B4C53E6F-001\tempDir-002\data\nodes\2\indices\dT7bmk_uQtqKf2wjfSn3Lg
  1> 
  1> 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:329) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:40]
  1> 	at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:558) ~[main/:?]
  1> 	at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:542) ~[main/:?]
  1> 	at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:652) [main/:?]
  1> 	at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:639) [main/:?]
  1> 	at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:634) [main/:?]
  1> 	at org.elasticsearch.indices.IndicesService.deleteUnassignedIndex(IndicesService.java:602) [main/:?]
  1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:263) [main/:?]
  1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:192) [main/:?]
  1> 	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [main/:?]
  1> 	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [main/:?]
  1> 	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [main/:?]
  1> 	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [main/:?]
  1> 	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [main/:?]
  1> 	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:576) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [main/:?]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

It also happened on 6.4, 6.x and master

I was not able to reproduce it on a Windows 2012 VM.
There's no evidence of similar failures in the past, nor recent changes to the 5.6
that are obviously related.

[elasticmachine]

Pinging @elastic/es-core-infra

[s1monw]

@jasontedor @bleskes any ideas here? I looked but it's really puzzeling

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

Relates #30962

[tvernum]

It's failed again.

I'm going to mute it on Windows unless someone says they need to capture more failure logs.