[4.2][SE-0202][stdlib] Random unification (#16495)

* Add shims for stdlib random

Initial random api

Use C syscall for I/O

1. Fixed an issue where integers would would result in an infinite loop if they were unsigned, or signed integers always returning negative numbers.
2. Fixed an issue with Bool initialization

Add shuffle functions

Add documentation to Random API

Fix a few typos within the documentation

Fixes more typos

Also states that the range for floating points is from 0 to 1 inclusive

Update API to reflect mailing list discussions

Remove unnecessary import

Make sure not to return upperBound on Range

Use SecRandomCopyBytes on older macOS

Update API to match mailing list discussion, add tests

Added pick(_:) to collection
Added random(in:using:) to Randomizable
Added tests

Fix typo in Randomizable documentation

Rename pick to sampling

Move sampling below random

Update docs

Use new Libc naming

Fix Random.swift with new Libc naming

Remove sampling

gybify signed integer creation

Make FloatingPoint.random exclusive

Refactor {Closed}Range.random

Fix FloatingPoint initialization

Precondition getting a random number from range

Fix some doc typos

Make .random a function

Update API to reflect discussion

Make .random a function
Remove .random() in favor of .random(in:) for all numeric types

Fix compile errors

Clean up _stdlib_random

Cleanup around API

Remove `.random()` requirement from `Collection`

Use generators

Optimize shuffle()

Thread safety for /dev/urandom

Remove {Closed}Range<BinaryFloatingPoint>.random()

Add Collection random requirement

Refactor _stdlib_random

Remove whitespace changes

Clean linux shim

Add shuffle and more tests

Provide finishing tests and suggestions

Remove refs to Countable ranges

Revert to checking if T is > UInt64

(cherry picked from commit d23d219)

* Add `_stdlib_random` for more platforms (#1)

* Remove refs to Countable ranges

* Add `_stdlib_random` for more platforms

* Use `getrandom` (if available) for Android, Cygwin

* Reorder the `_stdlib_random` functions

* Also include <features.h> on Linux

* Add `#error TODO` in `_stdlib_random` for Windows

* Colon after Fatal Error

Performance improvement for Random

gybify ranges

Fix typo in 'basic random numbers'
Add _stdlib_random as a testable method

Switch to generic constraints

Hopefully link against bcrypt

Fix some implementation details

1. Uniform distribution is now uniform
2. Apply Jens' method for uniform floats

Fix a lineable attribute

(cherry picked from commit a5df0ef)

* Revise documentation, add benchmarks (#3)

* [stdlib] Revise documentation for new random APIs

* [stdlib] Fix constraints on random integer generation

* [test] Isolate failing Random test

* [benchmark] Add benchmarks for new random APIs

Fix Float80 test

Value type generators

random -> randomElement

Fix some docs

One more doc fix

Doc fixes & bool fix

Use computed over explicit

(cherry picked from commit f146d17)

* Consolidate `_stdlib_random` functions (#2)

* Use the `__has_include` and `GRND_RANDOM` macros

* Use `getentropy` instead of `getrandom`

* Use `std::min` from the <algorithm> header

* Move `#if` out of the `_stdlib_random` function

* Use `getrandom` with "/dev/urandom" fallback

* Use `#pragma comment` to import "Bcrypt.lib"

* <https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp>
* <https://clang.llvm.org/docs/UsersManual.html#microsoft-extensions>

* Use "/dev/urandom" instead of `SecRandomCopyBytes`

* Use `swift::StaticMutex` for shared "/dev/urandom"

* Add `getrandom_available`; use `O_CLOEXEC` flag

Add platform impl docs

Update copyrights

Fix docs

Add _stdlib_random test

Update _stdlib_random test

Add missing &

Notice about _stdlib_random

Fix docs

Guard on upperBound = 0

Test full range of 8 bit integers

Remove some gyb

Clean up integerRangeTest

Remove FixedWidthInteger constraint

Use arc4random universally

Fix randomElement

Constrain shuffle to RandomAccessCollection

warning instead of error

Move Apple's implementation

Fix failing test on 32 bit systems

(cherry picked from commit b65d0c1)

* [stdlib] Add & implement Random._fill(bytes:) requirement

(cherry picked from commit 12a2b32)

* [stdlib] Random: don't randomize FixedWidthIntegers by overwriting their raw memory

Custom FixedWidthInteger types may not support this.

Introduce a new (non-public) FixedWidthInteger requirement for generating random values; implement it using &<</+ in the generic case, and specialize it using RandomNumberGenerator._fill(bytes) for the builtin types.

(cherry picked from commit 54b3b8b)

* [test] Move Random tests under validation-test

StdlibCollectionUnittest is not available in smoke tests.

(cherry picked from commit c03fe15)

* [test] Non-determinism down to 0.999999 (1-in-a-million) rather than 0.999.

With the number of tests Swift does, this had a relatively high chance to fail
regularly somewhere.

Also, rejecting the lower tail means rejecting things that are perfectly
uniform, which I don't think should be the purpose of this test.

* [test] Float80 only exists on some platforms.

Author: lorentey

benchmark/CMakeLists.txt

@@ -122,6 +122,8 @@ set(SWIFT_BENCH_MODULES
     single-source/Queue
     single-source/RC4
     single-source/RGBHistogram
+    single-source/RandomShuffle
+    single-source/RandomValues
     single-source/RangeAssignment
     single-source/RangeIteration
     single-source/RangeReplaceableCollectionPlusDefault

benchmark/single-source/RandomShuffle.swift

@@ -0,0 +1,64 @@
+//===--- RandomShuffle.swift ----------------------------------------------===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2018 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+
+import TestsUtils
+
+//
+// Benchmark that shuffles arrays of integers. Measures the performance of
+// shuffling large arrays.
+//
+
+public let RandomShuffle = [
+  BenchmarkInfo(name: "RandomShuffleDef", runFunction: run_RandomShuffleDef,
+    tags: [.api], setUpFunction: setup_RandomShuffle),
+  BenchmarkInfo(name: "RandomShuffleLCG", runFunction: run_RandomShuffleLCG,
+    tags: [.api], setUpFunction: setup_RandomShuffle),
+]
+
+/// A linear congruential PRNG.
+struct LCRNG: RandomNumberGenerator {
+  private var state: UInt64
+  
+  init(seed: Int) {
+    state = UInt64(truncatingIfNeeded: seed)
+    for _ in 0..<10 { _ = next() }
+  }
+  
+  mutating func next() -> UInt64 {
+    state = 2862933555777941757 &* state &+ 3037000493
+    return state
+  }
+}
+
+var numbers = Array(0...3_000_000)
+
+@inline(never)
+func setup_RandomShuffle() {
+  _ = numbers.count
+}
+
+@inline(never)
+public func run_RandomShuffleDef(_ N: Int) {
+  for _ in 0 ..< N {
+    numbers.shuffle()
+    blackHole(numbers.first!)
+  }
+}
+
+@inline(never)
+public func run_RandomShuffleLCG(_ N: Int) {
+  var generator = LCRNG(seed: 0)
+  for _ in 0 ..< N {
+    numbers.shuffle(using: &generator)
+    blackHole(numbers.first!)
+  }
+}

benchmark/single-source/RandomValues.swift

@@ -0,0 +1,87 @@
+//===--- RandomValues.swift -----------------------------------------------===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2018 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+
+import TestsUtils
+
+//
+// Benchmark generating lots of random values. Measures the performance of
+// the default random generator and the algorithms for generating integers
+// and floating-point values.
+//
+
+public let RandomValues = [
+  BenchmarkInfo(name: "RandomIntegersDef", runFunction: run_RandomIntegersDef, tags: [.api]),
+  BenchmarkInfo(name: "RandomIntegersLCG", runFunction: run_RandomIntegersLCG, tags: [.api]),
+  BenchmarkInfo(name: "RandomDoubleDef", runFunction: run_RandomDoubleDef, tags: [.api]),
+  BenchmarkInfo(name: "RandomDoubleLCG", runFunction: run_RandomDoubleLCG, tags: [.api]),
+]
+
+/// A linear congruential PRNG.
+struct LCRNG: RandomNumberGenerator {
+  private var state: UInt64
+  
+  init(seed: Int) {
+    state = UInt64(truncatingIfNeeded: seed)
+    for _ in 0..<10 { _ = next() }
+  }
+  
+  mutating func next() -> UInt64 {
+    state = 2862933555777941757 &* state &+ 3037000493
+    return state
+  }
+}
+
+@inline(never)
+public func run_RandomIntegersDef(_ N: Int) {
+  for _ in 0 ..< N {
+    var x = 0
+    for _ in 0 ..< 100_000 {
+      x &+= Int.random(in: 0...10_000)
+    }
+    blackHole(x)
+  }
+}
+
+@inline(never)
+public func run_RandomIntegersLCG(_ N: Int) {
+  for _ in 0 ..< N {
+    var x = 0
+    var generator = LCRNG(seed: 0)
+    for _ in 0 ..< 100_000 {
+      x &+= Int.random(in: 0...10_000, using: &generator)
+    }
+    CheckResults(x == 498214315)
+  }
+}
+
+@inline(never)
+public func run_RandomDoubleDef(_ N: Int) {
+  for _ in 0 ..< N {
+    var x = 0.0
+    for _ in 0 ..< 100_000 {
+      x += Double.random(in: -1000...1000)
+    }
+    blackHole(x)
+  }
+}
+
+@inline(never)
+public func run_RandomDoubleLCG(_ N: Int) {
+  for _ in 0 ..< N {
+    var x = 0.0
+    var generator = LCRNG(seed: 0)
+    for _ in 0 ..< 100_000 {
+      x += Double.random(in: -1000...1000, using: &generator)
+    }
+    blackHole(x)
+  }
+}

benchmark/utils/main.swift

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
+// Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -110,6 +110,8 @@ import ProtocolDispatch2
 import Queue
 import RC4
 import RGBHistogram
+import RandomShuffle
+import RandomValues
 import RangeAssignment
 import RangeIteration
 import RangeReplaceableCollectionPlusDefault
@@ -264,6 +266,8 @@ registerBenchmark(QueueGeneric)
 registerBenchmark(QueueConcrete)
 registerBenchmark(RC4Test)
 registerBenchmark(RGBHistogram)
+registerBenchmark(RandomShuffle)
+registerBenchmark(RandomValues)
 registerBenchmark(RangeAssignment)
 registerBenchmark(RangeIteration)
 registerBenchmark(RangeReplaceableCollectionPlusDefault)

stdlib/public/SwiftShims/LibcShims.h

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
+// Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -149,6 +149,10 @@ __swift_uint32_t _stdlib_cxx11_mt19937(void);
 SWIFT_RUNTIME_STDLIB_INTERNAL
 __swift_uint32_t _stdlib_cxx11_mt19937_uniform(__swift_uint32_t upper_bound);
 
+// Random number for stdlib
+SWIFT_RUNTIME_STDLIB_INTERNAL
+void _stdlib_random(void *buf, __swift_size_t nbytes);
+
 // Math library functions
 static inline SWIFT_ALWAYS_INLINE
 float _stdlib_remainderf(float _self, float _other) {

stdlib/public/core/Bool.swift

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
+// Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -86,6 +86,28 @@ public struct Bool {
   public init(_ value: Bool) {
     self = value
   }
+
+  /// Returns a random Boolean value
+  ///
+  /// - Parameter generator: The random number generator to use when getting a
+  ///   random Boolean.
+  /// - Returns: A random Boolean value.
+  @inlinable
+  public static func random<T: RandomNumberGenerator>(
+    using generator: inout T
+  ) -> Bool {
+    return (generator.next() >> 17) & 1 == 0
+  }
+  
+  /// Returns a random Boolean value
+  ///
+  /// - Returns: A random Boolean value.
+  ///
+  /// This uses the standard library's default random number generator.
+  @inlinable
+  public static func random() -> Bool {
+    return Bool.random(using: &Random.default)
+  }
 }
 
 extension Bool : _ExpressibleByBuiltinBooleanLiteral, ExpressibleByBooleanLiteral {

stdlib/public/core/CMakeLists.txt

@@ -2,7 +2,7 @@
 #
 # This source file is part of the Swift.org open source project
 #
-# Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
+# Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors
 # Licensed under Apache License v2.0 with Runtime Library Exception
 #
 # See https://swift.org/LICENSE.txt for license information
@@ -97,6 +97,7 @@ set(SWIFTLIB_ESSENTIAL
   Policy.swift
   PrefixWhile.swift
   Print.swift
+  Random.swift
   RandomAccessCollection.swift
   Range.swift
   RangeReplaceableCollection.swift

stdlib/public/core/Collection.swift

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
+// Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -803,6 +803,25 @@ public protocol Collection: Sequence where SubSequence: Collection {
   ///   `endIndex`.
   func formIndex(after i: inout Index)
 
+  /// Returns a random element of the collection, using the given generator as
+  /// a source for randomness.
+  ///
+  /// You use this method to select a random element from a collection when you
+  /// are using a custom random number generator. For example, call
+  /// `randomElement(using:)` to select a random element from an array of names.
+  ///
+  ///     let names = ["Zoey", "Chloe", "Amani", "Amaia"]
+  ///     let randomName = names.randomElement(using: &myGenerator)!
+  ///     // randomName == "Amani" (maybe)
+  ///
+  /// - Parameter generator: The random number generator to use when choosing
+  ///   a random element.
+  /// - Returns: A random element from the collection. If the collection is
+  ///   empty, the method returns `nil`.
+  func randomElement<T: RandomNumberGenerator>(
+    using generator: inout T
+  ) -> Element?
+
   @available(*, deprecated, message: "all index distances are now of type Int")
   typealias IndexDistance = Int
 }
@@ -1016,6 +1035,54 @@ extension Collection {
     return count
   }
 
+  /// Returns a random element of the collection, using the given generator as
+  /// a source for randomness.
+  ///
+  /// You use this method to select a random element from a collection when you
+  /// are using a custom random number generator. For example, call
+  /// `randomElement(using:)` to select a random element from an array of names.
+  ///
+  ///     let names = ["Zoey", "Chloe", "Amani", "Amaia"]
+  ///     let randomName = names.randomElement(using: &myGenerator)!
+  ///     // randomName == "Amani" (maybe)
+  ///
+  /// - Parameter generator: The random number generator to use when choosing
+  ///   a random element.
+  /// - Returns: A random element from the collection. If the collection is
+  ///   empty, the method returns `nil`.
+  @inlinable
+  public func randomElement<T: RandomNumberGenerator>(
+    using generator: inout T
+  ) -> Element? {
+    guard !isEmpty else { return nil }
+    let random = generator.next(upperBound: UInt(count))
+    let index = self.index(
+      startIndex,
+      offsetBy: numericCast(random)
+    )
+    return self[index]
+  }
+
+  /// Returns a random element of the collection.
+  ///
+  /// For example, call `randomElement()` to select a random element from an
+  /// array of names.
+  ///
+  ///     let names = ["Zoey", "Chloe", "Amani", "Amaia"]
+  ///     let randomName = names.randomElement()!
+  ///     // randomName == "Amani" (perhaps)
+  ///
+  /// This method uses the default random generator, `Random.default`. The call
+  /// to `names.randomElement()` above is equivalent to calling
+  /// `names.randomElement(using: &Random.default)`.
+  ///
+  /// - Returns: A random element from the collection. If the collection is
+  ///   empty, the method returns `nil`.
+  @inlinable
+  public func randomElement() -> Element? {
+    return randomElement(using: &Random.default)
+  }
+
   /// Do not use this method directly; call advanced(by: n) instead.
   @inlinable
   @inline(__always)