Test errors on Windows 32bit

[devmotion]

Hi!

As already mentioned in #17 (comment) and #19 (comment), some tests fail only on Windows 32bit. The problem does not only occur for these PRs but also for the master branch.

I tried to track down this problem but I'm not sure what's exactly the underlying cause. What I found out so far:

• The problem was introduced when I added additional unconstrained tests in
  https://github.com/JuliaDiffEq/DelayDiffEq.jl/blob/master/test/unconstrained.jl#L3-L57
  during the refactoring of the DDE problems (which were moved to DiffEqProblemLibrary). After that change the tests on Windows 32bit failed for the first time: https://ci.appveyor.com/project/ChrisRackauckas/delaydiffeq-jl/build/1.0.77/job/re955uo86yqud8cq
• The additional constrained tests pass without errors
• My first guess was some inaccuracy in the fixed-point iteration (which seemed like the only major difference between these tests), so I replaced the calculation of the residuals with some simple stupid assignments (https://github.com/devmotion/DelayDiffEq.jl/commit/221ff7ae5e901f2cb127e20b77e369e4b9bbc6e5). However, the tests still passed locally but failed on Windows 32bit
• I printed sol.t, sol.u, and sol.errors for these additional constrained and unconstrained tests (https://github.com/devmotion/DelayDiffEq.jl/commit/ceefa53821e8ebefc885c0658f35d1a32e47c705), and ran the tests again, both locally and on Windows 32bit. I got the following output on my computer:

julia> Pkg.test("DelayDiffEq")
INFO: Computing test dependencies for DelayDiffEq...
INFO: No packages to install, update or remove                                                                                     
INFO: Testing DelayDiffEq                                                                                                          
Test Summary:           | Pass  Total
Discontinuity Tree Test |    1      1
  4.993588 seconds (550.96 k allocations: 30.189 MiB, 0.30% gc time)
sol.t = [0.0, 0.1, 0.369144, 0.574108, 0.725764, 0.833981, 0.907633, 0.954618, 0.981867, 0.995353, 1.0, 1.00139, 1.0153, 1.15436, 2.0, 3.0, 3.28148, 3.35698, 3.52603, 3.65314, 3.81, 3.95676, 4.10746, 4.25055, 4.38703, 4.50295, 4.6409, 4.79308, 4.97254, 5.20421, 5.45049, 5.65683, 5.86996, 6.06915, 6.26313, 6.44645, 6.61885, 6.74856, 6.86535, 7.00722, 7.16207, 7.34754, 7.59843, 7.82791, 8.04062, 8.24993, 8.44994, 8.64243, 8.82501, 8.99568, 9.14872, 9.28259, 9.44973, 9.62997, 9.8607, 10.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998609, 0.984703, 0.845636, 1.11022e-16, -0.5, -0.464102, -0.443863, -0.385907, -0.33314, -0.260523, -0.188274, -0.11314, -0.0438535, 0.0181178, 0.0662641, 0.116931, 0.16321, 0.203556, 0.231992, 0.234213, 0.216656, 0.183803, 0.143529, 0.0992303, 0.0559762, 0.0165551, -0.011077, -0.0338119, -0.058078, -0.0797574, -0.0985111, -0.111028, -0.110233, -0.100506, -0.0843359, -0.0646734, -0.0436413, -0.0232667, -0.0049751, 0.0100674, 0.0217497, 0.0339833, 0.0438881, 0.0513028, 0.0529556]
sol.errors = Dict(:l∞=>2.96934e-5,:final=>1.78371e-5,:l2=>1.35508e-5)
sol.t = [0.0, 0.1, 0.149934, 0.179314, 0.194233, 0.2, 0.201641, 0.218051, 0.273502, 0.306588, 0.324284, 0.331958, 0.333333, 0.333862, 0.339152, 0.39205, 0.4, 0.409802, 0.507823, 0.533333, 0.6, 0.666667, 0.712008, 0.733333, 0.783429, 0.81763, 0.865753, 0.866667, 0.875806, 0.885145, 0.899284, 0.914972, 0.932586, 0.949364, 0.972833, 0.998694, 1.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998359, 0.981949, 0.926498, 0.893412, 0.875716, 0.868042, 0.866667, 0.865609, 0.855029, 0.749233, 0.733329, 0.713773, 0.523496, 0.475552, 0.357774, 0.25328, 0.190706, 0.164046, 0.108254, 0.07548, 0.0363202, 0.0356527, 0.0291302, 0.0227476, 0.0136143, 0.00420783, -0.00547466, -0.0138657, -0.024304, -0.0341455, -0.0345982]
sol.errors = Dict(:l∞=>4.03291e-6,:final=>1.46663e-6,:l2=>2.22825e-6)
Test Summary:        | Pass  Total
Constrained Timestep |   10     10
 27.103416 seconds (14.49 M allocations: 1.939 GiB, 2.65% gc time)
sol.t = [0.0, 1.0e-6, 1.1e-5, 0.000111, 0.001111, 0.011111, 0.111111, 0.377703, 0.580532, 0.730432, 0.837233, 0.909777, 0.955924, 0.982569, 0.995653, 1.0, 1.00132, 1.01453, 1.14666, 2.0, 3.0, 3.26511, 3.33927, 3.50374, 3.62911, 3.78416, 3.93023, 4.08103, 4.22455, 4.36281, 4.48381, 4.6155, 4.76682, 4.9402, 5.15894, 5.43584, 5.63522, 5.85524, 6.05396, 6.25046, 6.43464, 6.60866, 6.74208, 6.86279, 7.00486, 7.16185, 7.34824, 7.60133, 7.83111, 8.04468, 8.2543, 8.45465, 8.64726, 8.82985, 9.00035, 9.15292, 9.28823, 9.45597, 9.63774, 9.87158, 10.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998679, 0.985466, 0.853337, 1.11022e-16, -0.5, -0.467964, -0.448955, -0.394426, -0.343608, -0.272909, -0.201494, -0.126241, -0.0561746, 0.00749744, 0.0586436, 0.108196, 0.155992, 0.197464, 0.228516, 0.234825, 0.219217, 0.18646, 0.146831, 0.102202, 0.0587451, 0.018811, -0.0097538, -0.0333402, -0.0577081, -0.0797298, -0.0985655, -0.111087, -0.110144, -0.100245, -0.0839407, -0.0641743, -0.0431001, -0.0227317, -0.00449233, 0.0104579, 0.0222076, 0.0343852, 0.0442345, 0.0515062, 0.0529548]
sol.errors = Dict(:l∞=>3.33279e-5,:final=>1.8635e-5,:l2=>1.40649e-5)
sol.t = [0.0, 1.0e-6, 1.1e-5, 0.000111, 0.001111, 0.011111, 0.111111, 0.156734, 0.183003, 0.195837, 0.2, 0.201278, 0.214062, 0.271031, 0.305181, 0.323593, 0.331714, 0.333333, 0.333933, 0.339929, 0.399889, 0.4, 0.401112, 0.412228, 0.523388, 0.533333, 0.546797, 0.6, 0.666667, 0.712008, 0.733333, 0.783169, 0.817419, 0.865489, 0.866667, 0.87844, 0.888516, 0.904293, 0.920763, 0.938668, 0.958127, 0.982948, 1.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998722, 0.985938, 0.928969, 0.894819, 0.876407, 0.868286, 0.866667, 0.865467, 0.853475, 0.733556, 0.733333, 0.731111, 0.708953, 0.494169, 0.475554, 0.450694, 0.357776, 0.253282, 0.190708, 0.164048, 0.108521, 0.0756704, 0.0365139, 0.0356535, 0.0273022, 0.0205131, 0.0105298, 0.000924303, -0.00860783, -0.0179362, -0.0283543, -0.0345979]
sol.errors = Dict(:l∞=>1.85086e-6,:final=>1.15435e-6,:l2=>1.04234e-6)
Standard tests complete. Onto idxs tests
  7.321181 seconds (13.41 M allocations: 479.789 MiB, 1.99% gc time)
  5.287681 seconds (9.22 M allocations: 408.107 MiB, 2.27% gc time)
Test Summary:          | Pass  Broken  Total
Unconstrained Timestep |   14       2     16
 40.970641 seconds (48.47 M allocations: 3.004 GiB, 2.72% gc time)
Test Summary: | Pass  Total
Events        |    4      4
  8.789850 seconds (4.54 M allocations: 597.395 MiB, 2.74% gc time)
Test Summary: | Pass  Total
Units         |    6      6
 25.956555 seconds (55.17 M allocations: 1.900 GiB, 3.36% gc time)
Test Summary: | Pass  Total
Unique Times  |    2      2
  4.135222 seconds (2.09 M allocations: 285.094 MiB, 2.50% gc time)
INFO: DelayDiffEq tests passed
INFO: No packages to install, update or remove

The same tests on Windows 32bit yield (among others)

sol.t = [0.0, 0.1, 0.369144, 0.574108, 0.725764, 0.833981, 0.907633, 0.954618, 0.981867, 0.995353, 1.0, 1.00139, 1.0153, 1.15436, 2.0, 3.0, 3.28148, 3.35698, 3.52603, 3.65314, 3.81, 3.95676, 4.10746, 4.25055, 4.38703, 4.50295, 4.6409, 4.79308, 4.97254, 5.20421, 5.45049, 5.65683, 5.86996, 6.06915, 6.26313, 6.44645, 6.61885, 6.74856, 6.86535, 7.00722, 7.16207, 7.34754, 7.59843, 7.82791, 8.04062, 8.24993, 8.44994, 8.64243, 8.82501, 8.99568, 9.14872, 9.28259, 9.44973, 9.62997, 9.8607, 10.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998609, 0.984703, 0.845636, 1.11022e-16, -0.5, -0.464102, -0.443863, -0.385907, -0.33314, -0.260523, -0.188274, -0.11314, -0.0438535, 0.0181178, 0.0662641, 0.116931, 0.16321, 0.203556, 0.231992, 0.234213, 0.216656, 0.183803, 0.143529, 0.0992303, 0.0559762, 0.0165551, -0.011077, -0.0338119, -0.058078, -0.0797574, -0.0985111, -0.111028, -0.110233, -0.100506, -0.0843359, -0.0646734, -0.0436413, -0.0232667, -0.0049751, 0.0100674, 0.0217497, 0.0339833, 0.0438881, 0.0513028, 0.0529556]
sol.errors = Dict(:final=>1.78371e-5,:l2=>1.35508e-5,:l∞=>2.96934e-5)
sol.t = [0.0, 0.1, 0.149934, 0.179314, 0.194233, 0.2, 0.201641, 0.218051, 0.273502, 0.306588, 0.324284, 0.331958, 0.333333, 0.333862, 0.339152, 0.39205, 0.4, 0.409802, 0.507823, 0.533333, 0.6, 0.666667, 0.712008, 0.733333, 0.783429, 0.81763, 0.865753, 0.866667, 0.875806, 0.885145, 0.899284, 0.914972, 0.932586, 0.949364, 0.972833, 0.998694, 1.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998359, 0.981949, 0.926498, 0.893412, 0.875716, 0.868042, 0.866667, 0.865609, 0.855029, 0.749233, 0.733329, 0.713773, 0.523496, 0.475552, 0.357774, 0.25328, 0.190706, 0.164046, 0.108254, 0.07548, 0.0363202, 0.0356527, 0.0291302, 0.0227476, 0.0136143, 0.00420783, -0.00547466, -0.0138657, -0.024304, -0.0341455, -0.0345982]
sol.errors = Dict(:final=>1.46663e-6,:l2=>2.22825e-6,:l∞=>4.03291e-6)
Test Summary:        | Pass  Total
Constrained Timestep |   10     10
 92.728570 seconds (14.67 M allocations: 1023.284 MiB, 0.70% gc time)
sol.t = [0.0, 0.0001, 0.0011, 0.0111, 0.1111, 0.377695, 0.580526, 0.730427, 0.83723, 0.909775, 0.955922, 0.982568, 0.995653, 1.0, 1.00132, 1.01453, 1.14667, 2.0, 3.0, 3.26511, 3.33927, 3.50374, 3.62911, 3.78416, 3.93023, 4.08103, 4.22455, 4.36281, 4.48381, 4.6155, 4.76682, 4.9402, 5.15894, 5.43584, 5.63522, 5.85524, 6.05396, 6.25046, 6.43464, 6.60866, 6.74208, 6.86279, 7.00486, 7.16185, 7.34824, 7.60133, 7.83111, 8.04468, 8.2543, 8.45465, 8.64726, 8.82985, 9.00035, 9.15292, 9.28823, 9.45597, 9.63774, 9.87158, 10.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998679, 0.985465, 0.853329, 1.11022e-16, -0.5, -0.467964, -0.448955, -0.394426, -0.343608, -0.272909, -0.201494, -0.126241, -0.0561746, 0.00749744, 0.0586436, 0.108196, 0.155992, 0.197464, 0.228516, 0.234825, 0.219217, 0.18646, 0.146831, 0.102202, 0.0587451, 0.018811, -0.0097538, -0.0333402, -0.0577081, -0.0797298, -0.0985655, -0.111087, -0.110144, -0.100245, -0.0839407, -0.0641743, -0.0431001, -0.0227317, -0.00449233, 0.0104579, 0.0222076, 0.0343852, 0.0442345, 0.0515062, 0.0529548]
sol.errors = Dict(:final=>1.8635e-5,:l2=>1.43013e-5,:l∞=>3.33279e-5)
sol.t = [0.0, 0.0001, 0.0011, 0.0111, 0.1111, 0.156727, 0.182999, 0.195835, 0.2, 0.201279, 0.214065, 0.271034, 0.305183, 0.323594, 0.331715, 0.333333, 0.333933, 0.339928, 0.399881, 0.4, 0.401189, 0.413074, 0.531929, 0.533333, 0.538473, 0.58987, 0.6, 0.666667, 0.711969, 0.733333, 0.783166, 0.817407, 0.865494, 0.866667, 0.878392, 0.888456, 0.904206, 0.920664, 0.938568, 0.957985, 0.982786, 1.0]
sol.u = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.998721, 0.985935, 0.928966, 0.894817, 0.876406, 0.868285, 0.866667, 0.865468, 0.853477, 0.733571, 0.733333, 0.730957, 0.707271, 0.478178, 0.475555, 0.466, 0.374815, 0.357777, 0.253283, 0.190759, 0.164048, 0.108524, 0.0756807, 0.0365097, 0.0356528, 0.0273341, 0.0205515, 0.0105816, 0.000978592, -0.00855772, -0.0178725, -0.0282924, -0.0345987]
sol.errors = Dict(:final=>2.0367e-6,:l2=>1.25787e-6,:l∞=>2.3578e-6)
Unconstrained Timestep: Test Failed
  Expression: sol.errors[:l∞] < 2.0e-6
   Evaluated: 2.3578041580193942e-6 < 2.0e-6
Stacktrace:
 [1] include_from_node1(::String) at .\loading.jl:569
 [2] include at .\sysimg.jl:14
 [3] macro expansion at C:\Users\appveyor\.julia\v0.6\DelayDiffEq\test\runtests.jl:6 [inlined]
 [4] macro expansion at .\test.jl:860 [inlined]
 [5] macro expansion at .\util.jl:237 [inlined]
 [6] anonymous at .\<missing>:?
Unconstrained Timestep: Test Failed
  Expression: sol.errors[:final] < 2.0e-6
   Evaluated: 2.0367022636677556e-6 < 2.0e-6
Stacktrace:
 [1] include_from_node1(::String) at .\loading.jl:569
 [2] include at .\sysimg.jl:14
 [3] macro expansion at C:\Users\appveyor\.julia\v0.6\DelayDiffEq\test\runtests.jl:6 [inlined]
 [4] macro expansion at .\test.jl:860 [inlined]
 [5] macro expansion at .\util.jl:237 [inlined]
 [6] anonymous at .\<missing>:?
Standard tests complete. Onto idxs tests
  5.337208 seconds (13.41 M allocations: 319.968 MiB, 3.03% gc time)
  4.102716 seconds (9.22 M allocations: 252.775 MiB, 5.94% gc time)
Test Summary:          | Pass  Fail  Broken  Total
Unconstrained Timestep |   12     2       2     16

The complete log file can be found here. Interestingly, the constrained tests seem to generate the same sol.t, sol.u, and sol.errors; however, in both unconstrained tests already the first time step differs (1.0e-6 vs. 0.0001).

Thus my guess so far is that on Windows 32bit somehow a different initial time step is calculated, which causes all other discrepancies in the solution. However, I do not know why a different time step is calculated in the first place.

[devmotion]

As the hypothesis above suggests, not setting the initial time step dt = 0.1 in the constrained tests leads to different initial time steps also for the constrained tests (1.0e-06 vs. 0.0001). As a consequence, also the constrained tests fail (see https://ci.appveyor.com/project/devmotion/delaydiffeq-jl-p4v3l/build/job/fhu7jik5t33r1jy6).

[ChrisRackauckas]

As the hypothesis above suggests, not setting the initial time step dt = 0.1 in the constrained tests leads to different initial time steps also for the constrained tests (1.0e-06 vs. 0.0001).

That's really weird. Doesn't it suggest there's a difference due to OrdinaryDiffEq.jl and not DelayDiffEq.jl though?

[devmotion]

I'm not sure yet. At least I could not reproduce the difference with a simple ODE example. Moreover, I discovered that the initial time step in these examples is set to 1.0e-6 on my computer and 9.999999999999999e-5 on Windows 32bit, but still the first non-zero time point of the solution is 1.0e-6 on my computer and 0.0001 on Windows 32bit.

[devmotion]

Oh my god :D Of course 9.9999e-5 != 1.0e-6

[devmotion]

Fixed by SciML/OrdinaryDiffEq.jl@b70c49e

[devmotion]

Seems there is another problem on Windows 32bit that was not fixed by that commit, since now another test fails:
https://ci.appveyor.com/project/ChrisRackauckas/delaydiffeq-jl/build/1.0.96/job/l0ty4hq534tsxhep

[ChrisRackauckas]

Only 32-bit and only unconstrained tests with a really small difference? I'll take a quick look but that sounds like a nightmare to find haha.

[devmotion]

Yes, and it's only one single test.

[ChrisRackauckas]

Discussed in #22. The errors are fixed because they needed a higher bound anyways after the OrdinaryDiffEq changes. A lot of this may be due to compiler optimizations not be applied uniformly. Practical steps forward would be to reduce numerical errors all around via things like SciML/OrdinaryDiffEq.jl#93 and summation re-ordering. I'm closing this because I think the @fastmath tests show that the level of accuracy that was looked at here is just beyond our control, but we can focus on SciML/OrdinaryDiffEq.jl#93 instead. Feel free to pursue this further if you want though, and we can re-open if you find something.