Skip to content

Conversation

@kaiyaok2
Copy link

@kaiyaok2 kaiyaok2 commented Apr 14, 2022

Description
The tests in CachingObjectStorage.java dealing with multiple cached requests can fail nondeterministically when running in different JVMs. The issue is found using the command mvn edu.illinois:nondex-maven-plugin:1.1.2:nondex under the hdfs-connector directory after building dependencies.

Reason for Flakiness
The tests call Mockito.verify() to assure that the mock client calls the routine getObject() only once when same requests are cached. However, when running in different environments, these verifications may occassionally fail. The reason is that in CachingObjectStorage.getObject(), the code calls Cache.put() with a key-value pair to update the cache. The method will create a new key-value pair ( rather than replacing the old value with the input value) if the input key (hashcode-wise) is not equal to any keys in the cache. However, in the hashcode() method, the final object to be hashed is an Array, which preserves order. While the output of getDeclaredMethods() does not guarantee any order, the array for hashing (line 779 at CachingObjectStorage.java) may have nondeterministic element orders. As a result, the hashcode of two same keys can be different, leading to "duplicated" entries in the cache, and the mock client can then call getObject() more times than desired.

List of flaky tests in CachingObjectStorage.java

  • testPrepopulateCache()
  • testTwoGetsCached()
  • testTwoGets_Cached()
  • testTwoGets_Cached_StrongConsistency()
  • testTwoGets_Cached_StrongConsistency_ifMatch()
  • testTwoGets_Cached_StrongConsistency_ifMatch_changed()
  • testTwoDistinctGets_StrongConsistency()
  • testTwoDistinctGets_StrongConsistency_ETagChanges()

Sample Failure

Failed tests:
  CachingObjectStorageTest.testTwoGetsCached:243
mockClient.getObject(<any>);
Wanted 1 time:
-> at com.oracle.bmc.hdfs.caching.CachingObjectStorageTest.testTwoGetsCached(CachingObjectStorageTest.java:243)
But was 2 times.

Proposed Fix
Convert the array into a normal HashSet before hashing to avoid unwanted order dependency.

OCA signed
https://oca.opensource.oracle.com/api/v1/oca-requests/6710/documents/6730/download
(approved)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant