Fair comparison between Stan and Turing

[xukai92]

We should add settings via Stan.jl to make sure that the comparisons are between Stan.HMC vs Turing.HMC or Stan.NUTS vs Turing.NUTS.

[yebai]

[HMC] Sampling...100% Time: 0:02:17
[HMC] Finished with
  Running time        = 137.39043471199986;
  Accept rate         = 0.1215;
  #lf / sample        = 9.995;
  #evals / sample     = 9.996;
  pre-cond. diag mat  = [3.53885,9.83359,6.44391,14.54....
/=======================================================================
| Benchmark Result for >>> LDA <<<
|-----------------------------------------------------------------------
| Overview
|-----------------------------------------------------------------------
| Inference Engine  : HMC(2000, 0.025, 10)
| Time Used (s)     : 141.96591081200023
|   -> time by Stan : 2.96523
| Mem Alloc (bytes) : 51643383864
|-----------------------------------------------------------------------
| Turing Inference Result
|-----------------------------------------------------------------------
| >> phi[2] <<
| mean = [0.023,0.0,0.363,0.371,0.243]
|   -> Stan     = [0.396,0.337,0.255,0.003,0.009], diff = [0.373,0.337,0.107,0.367,0.235]
| >> phi[1] <<
| mean = [0.439,0.423,0.1,0.002,0.037]
|   -> Stan     = [0.007,0.003,0.281,0.406,0.303], diff = [0.432,0.42,0.181,0.404,0.267]
\=======================================================================

[yebai]

pre-cond. diag mat = [3.53885,9.83359,6.44391,14.54....

@xukai92 This should be an identity matrix for HMC. Seems HMC is still adapting parameters when n_adapt is set to 0. Any idea why?

PS. These running time numbers are produced using the same HMC sampler with fixed stepsize and integration steps, but without adaption.

[xukai92]

I think I apply pre-condition globally, i.e. to all HMC algorithms.

[yebai]

New running time

/=======================================================================
| Benchmark Result for >>> LDA-vec <<<
|-----------------------------------------------------------------------
| Overview
|-----------------------------------------------------------------------
| Inference Engine  : HMC(2000, 0.025, 10)
| Time Used (s)     : 57.871653644000006
|   -> time by Stan : 2.3655
| Mem Alloc (bytes) : 33152713975
|-----------------------------------------------------------------------
| Turing Inference Result
|-----------------------------------------------------------------------
| >> phi[2] <<
| mean = [0.299,0.252,0.427,0.005,0.017]
|   -> Stan     = [0.004,0.003,0.265,0.414,0.313], diff = [0.295,0.249,0.161,0.409,0.296]
| >> phi[1] <<
| mean = [0.015,0.014,0.0,0.567,0.404]
|   -> Stan     = [0.39,0.33,0.272,0.005,0.002], diff = [0.375,0.316,0.272,0.561,0.401]
\=======================================================================

[yebai]

/=======================================================================
| Benchmark Result for >>> LDA <<<
|-----------------------------------------------------------------------
| Overview
|-----------------------------------------------------------------------
| Inference Engine  : HMC(2000, 0.025, 10)
| Time Used (s)     : 96.22526677100001
|   -> time by Stan : 2.3655
| Mem Alloc (bytes) : 46339414770
|-----------------------------------------------------------------------
| Turing Inference Result
|-----------------------------------------------------------------------
| >> phi[2] <<
| mean = [0.015,0.007,0.325,0.369,0.284]
|   -> Stan     = [0.004,0.003,0.265,0.414,0.313], diff = [0.011,0.004,0.06,0.046,0.029]
| >> phi[1] <<
| mean = [0.432,0.371,0.173,0.014,0.01]
|   -> Stan     = [0.39,0.33,0.272,0.005,0.002], diff = [0.042,0.041,0.099,0.009,0.008]
\=======================================================================

[yebai]

I believe now we can make Turing programs much faster by writing better Julia code in the @model block.

[xukai92]

Awesome improvement!