Skip to content

Commit a7b430b

Browse files
committed
[SPARK-15354][FLAKY-TEST] TopologyAwareBlockReplicationPolicyBehavior.Peers in 2 racks
## What changes were proposed in this pull request? `TopologyAwareBlockReplicationPolicyBehavior.Peers in 2 racks` is failing occasionally: https://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.storage.TopologyAwareBlockReplicationPolicyBehavior&test_name=Peers+in+2+racks. This is because, when we generate 10 block manager id to test, they may all belong to the same rack, as the rack is randomly picked. This PR fixes this problem by forcing each rack to be picked at least once. ## How was this patch tested? N/A Author: Wenchen Fan <[email protected]> Closes #17624 from cloud-fan/test.
1 parent 924c424 commit a7b430b

File tree

1 file changed

+11
-2
lines changed

1 file changed

+11
-2
lines changed

core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,18 @@ class RandomBlockReplicationPolicyBehavior extends SparkFunSuite
7070
}
7171
}
7272

73+
/**
74+
* Returns a sequence of [[BlockManagerId]], whose rack is randomly picked from the given `racks`.
75+
* Note that, each rack will be picked at least once from `racks`, if `count` is greater or equal
76+
* to the number of `racks`.
77+
*/
7378
protected def generateBlockManagerIds(count: Int, racks: Seq[String]): Seq[BlockManagerId] = {
74-
(1 to count).map{i =>
75-
BlockManagerId(s"Exec-$i", s"Host-$i", 10000 + i, Some(racks(Random.nextInt(racks.size))))
79+
val randomizedRacks: Seq[String] = Random.shuffle(
80+
racks ++ racks.length.until(count).map(_ => racks(Random.nextInt(racks.length)))
81+
)
82+
83+
(0 until count).map { i =>
84+
BlockManagerId(s"Exec-$i", s"Host-$i", 10000 + i, Some(randomizedRacks(i)))
7685
}
7786
}
7887
}

0 commit comments

Comments
 (0)