End-to-end perf scenarios

[briancylui]

Performance Test

test\Microsoft.ML.Benchmarks\StochasticDualCoordinateAscentClassifierBench.cs

Results

.NET Core App 2.0 (Job=Default):

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.200
  [Host] : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT

Toolchain=InProcessToolchain

Method	Mean	Error	StdDev	AccuracyMacro	Gen 0	Gen 1	Gen 2	Allocated
TrainIris	340,686,491.15 ns	3,571,557.1056 ns	2,982,411.6534 ns	0.98	8625.0000	250.0000	62.5000	12596368 B
PredictIris	1,292,667.88 ns	25,756.8687 ns	64,618.9476 ns	0.98	25.3906	9.7656	1.9531	87571 B
PredictIrisBatchOf1	14.80 ns	0.2822 ns	0.2357 ns	0.98	0.0171	-	-	72 B
PredictIrisBatchOf2	15.14 ns	0.3011 ns	0.3468 ns	0.98	0.0171	-	-	72 B
PredictIrisBatchOf5	16.40 ns	0.3957 ns	0.5145 ns	0.98	0.0171	-	-	72 B

// * Legends *
  Mean          : Arithmetic mean of all measurements
  Error         : Half of 99.9% confidence interval
  StdDev        : Standard deviation of all measurements
  AccuracyMacro : Macro-average accuracy of the model
  Gen 0         : GC Generation 0 collects per 1k Operations
  Gen 1         : GC Generation 1 collects per 1k Operations
  Gen 2         : GC Generation 2 collects per 1k Operations
  Allocated     : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ns          : 1 Nanosecond (0.000000001 sec)

.NET Core App 3.0 (Job=Default):

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
  [Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT

Toolchain=InProcessToolchain

Method	Mean	Error	StdDev	AccuracyMacro	Gen 0	Gen 1	Gen 2	Allocated
TrainIris	340,069,667.23 ns	4,074,874.9274 ns	3,612,270.1309 ns	0.98	10187.5000	312.5000	62.5000	14416239 B
PredictIris	1,163,328.54 ns	21,833.5386 ns	21,443.4523 ns	0.98	27.3438	13.6719	1.9531	89840 B
PredictIrisBatchOf1	13.58 ns	0.1441 ns	0.1277 ns	0.98	0.0171	-	-	72 B
PredictIrisBatchOf2	13.58 ns	0.1992 ns	0.1863 ns	0.98	0.0171	-	-	72 B
PredictIrisBatchOf5	13.63 ns	0.2046 ns	0.1598 ns	0.98	0.0171	-	-	72 B

// * Legends *
  Mean          : Arithmetic mean of all measurements
  Error         : Half of 99.9% confidence interval
  StdDev        : Standard deviation of all measurements
  AccuracyMacro : Macro-average accuracy of the model
  Gen 0         : GC Generation 0 collects per 1k Operations
  Gen 1         : GC Generation 1 collects per 1k Operations
  Gen 2         : GC Generation 2 collects per 1k Operations
  Allocated     : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ns          : 1 Nanosecond (0.000000001 sec)

[helloguo]

@briancylui , any idea which CpuMath APIs are used in this example? And what's the percentage of the execution time spending on these CpuMath APIs vs overall execution time ?

[briancylui]

@helloguo:
(1) While I don't have this specific info, SdcaL1UpdatedU and SdcaL1UpdateSU could probably be used since Sdca is short for StochasticDualCoordinateAscent. Meanwhile, the code here (https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs) could provide some context.

(2) Unfortunately I don't have this specific info, but it will be useful to know about it.

[eerhardt]

Thanks for the results, @briancylui. It's good to see the time drop, while keeping the same accuracy. However, one concern I have is that on .NET Core 3.0, it looks like we are allocating MORE memory for both TrainIris and PredictIris.

For one more comparison, can you run the same perf test against .NET Core 2.1? .NET Core 2.0 is a bit old by now, and a bunch of perf changes came in 2.1.