Skip to content

Commit 57f7046

Browse files
committed
[test] Non-determinism down to 0.999999 (1-in-a-million) rather than 0.999.
With the number of tests Swift does, this had a relatively high chance to fail regularly somewhere. Also, rejecting the lower tail means rejecting things that are perfectly uniform, which I don't think should be the purpose of this test.
1 parent 7e4a16c commit 57f7046

File tree

1 file changed

+8
-3
lines changed

1 file changed

+8
-3
lines changed

validation-test/stdlib/Random.swift

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -235,13 +235,18 @@ RandomTests.test("different random number generators") {
235235
func chi2Test(_ samples: [Double]) -> Bool {
236236
precondition(samples.count == 50, "confidence interval requires 50 samples")
237237
let expected = samples.reduce(0, +) / Double(samples.count)
238-
let cvLow = 23.983 // 0.1% with a degree of freedom of (50 - 1)
239-
let cvHigh = 85.351 // 99.9% with a degree of freedom of (50 - 1)
238+
// Right tail for 0.0001% (1e-6) with a degree of freedom of (50 -
239+
// 1). With hundreds of builds a day, this has to be very low to not get too
240+
// many spurious failures, but obvious problems should still be detected
241+
// (e.g. an off-by-one that means samples[0] == 0 will result in chi2 =
242+
// (0-expected)**2/expected + ... > expected, so as long as we've generated
243+
// more than samples.count * cvHigh (~5500) values, we'll catch it).
244+
let cvHigh = 111.1
240245
let chi2 = samples.map {
241246
(($0 - expected) * ($0 - expected)) / expected
242247
}.reduce(0, +)
243248

244-
if chi2 < cvLow || chi2 > cvHigh {
249+
if chi2 > cvHigh {
245250
return false
246251
}else {
247252
return true

0 commit comments

Comments
 (0)