Surprising behavior for default/interactive thread pools

[samtkaplan]

By default, it appears that work is assigned to the interactive thread pool (if it exists) rather than the default thread pool. The expected behavior is that, by default, work gets assigned to the default thread pool, and assignment to the interactive thread pool requires either appropriate use of the interactive REPL, or use of Threads.@spawn :interactive. Please also see a corresponding discourse post.

Here are some examples that reproduce the problem:

Example 1

julia -t 1,1 -e 'print(stdout, "$(Threads.threadpool())\n")'

outputs:

interactive

Example 3

julia -t 1,1 -e 'Threads.threadid() in Threads.threadpooltids(:interactive) ? println("interactive") : println("default"); println(isinteractive())'

outputs

interactive
false

Example 4

julia -t 3,1 -E 'Threads.threadpool()'
:interactive

[vchuravy]

That is expected behavior the root task is an interactive task. This is necessary since currently it processes all IO (outside of @threads).

[carstenbauer]

That is expected behavior the root task is an interactive task. This is necessary since currently it processes all IO (outside of @threads).

That's what I thought but

julia -t 3 -E 'Threads.threadpool()'
:default

So it isn't always an interactive task. In fact, assuming that most people don't start Julia with explicit interactive threads, it typically isn't.

[carstenbauer]

And what the OP forgot to mention, the documentation here is simply wrong:

julia -t 3,1 -E 'Threads.threadpool()'
:interactive # the doc says :default here

This certainly needs to be corrected.

[vtjnash]

I think someone was working on a docfix for that in a PR recently

[carstenbauer]

I can't find a PR for this. I think you might be confusing it with #53117.

Apart from the docs, what about my first point? Seems at least a bit confusing to me that the root thread is in different thread pools depending on how I start Julia.

[samtkaplan]

From my point of view (a simple user), it is confusing that by default code does not use a thread from the default thread pool (assuming that I started Julia with at least one interactive thread). In other words, the statement "by default code does not use a thread from the default thread pool" sounds wrong by virtue of its construction. Not sure if that feed-back can help with the eventual change to the docs, but hope it helps :)

[carstenbauer]

The fact that the main thread is sometimes in :default and sometimes in :interactive, depending on if one starts Julia explicitly with at least one interactive thread, seems to be causing problems (see https://discourse.julialang.org/t/what-is-julia-doing-with-your-threads/110052/20?u=carstenbauer and the related issue #53269).

What can (and should) be done right away is to fix the documentation and document things accurately (I might be able to make a PR tonight).

But I'm still curious what's the reasoning behind this behavior. Why not make the main thread always :interactive?

[PetrKryslUCSD]

I must be misunderstanding something here. This is what I get when running as a batch and also as interactive:

>>>>>>>>>>>>>>> BATCH <<<<<<<<<<<<<<<<<<<<<<
PS C:\Users\pkonl\Documents\00WIP\FinEtoolsHeatDiff.jl\examples> julia -t 5 .\steady_state\3-d\mwe_tasks.jl  
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 3, thread 5, default: Spawned 5.99999e-02
[ Info: Chunk 2, thread 2, default: Spawned 6.39999e-02
[ Info: Chunk 1, thread 3, default: Spawned 4.72000e-01
[ Info: Chunk 4, thread 1, default: Spawned 4.74000e-01
[ Info: 4: Finished 1.77100e+00
[ Info: 3: Finished 1.77200e+00
[ Info: 2: Finished 1.77300e+00
[ Info: 1: Finished 1.77300e+00
[ Info: Finished 1.77400e+00
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 4, thread 1, default: Spawned 0.00000e+00
[ Info: Chunk 3, thread 4, default: Spawned 9.99928e-04
[ Info: Chunk 2, thread 3, default: Spawned 9.99928e-04
[ Info: Chunk 1, thread 2, default: Spawned 9.99928e-04
[ Info: 4: Finished 1.29500e+00
[ Info: 3: Finished 1.29600e+00
[ Info: 1: Finished 1.29700e+00
[ Info: 2: Finished 1.29900e+00
[ Info: Finished 1.29900e+00
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 3, thread 3, default: Spawned 0.00000e+00
[ Info: Chunk 4, thread 1, default: Spawned 0.00000e+00
[ Info: Chunk 2, thread 4, default: Spawned 0.00000e+00
[ Info: Chunk 1, thread 2, default: Spawned 0.00000e+00
[ Info: 1: Finished 1.28900e+00
[ Info: 3: Finished 1.28900e+00
[ Info: 4: Finished 1.29300e+00
[ Info: 2: Finished 1.29600e+00
[ Info: Finished 1.29700e+00
extrema(ts) = (1.29906e+00, 1.30095e+00)


>>>>>>>>>>>>> INTERACTIVE <<<<<<<<<<<<<<<<<<<
PS C:\Users\pkonl\Documents\00WIP\FinEtoolsHeatDiff.jl\examples> julia -t5
 
julia> include(raw"C:\Users\pkonl\Documents\00WIP\FinEtoolsHeatDiff.jl\examples\steady_state\3-d\mwe_tasks.jl")
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 4, thread 1, default: Spawned 6.00002e-02
[ Info: Chunk 1, thread 3, default: Spawned 4.86000e-01
[ Info: Chunk 2, thread 2, default: Spawned 4.89000e-01
[ Info: Chunk 3, thread 4, default: Spawned 6.40001e-02
[ Info: 4: Finished 1.85500e+00
[ Info: 3: Finished 3.14000e+00
[ Info: 1: Finished 3.14100e+00
[ Info: 2: Finished 3.14400e+00
[ Info: Finished 3.14500e+00
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 1, thread 1, default: Spawned 0.00000e+00
[ Info: Chunk 2, thread 3, default: Spawned 0.00000e+00
[ Info: Chunk 4, thread 5, default: Spawned 0.00000e+00
[ Info: Chunk 3, thread 4, default: Spawned 0.00000e+00
[ Info: 1: Finished 1.30800e+00
[ Info: 4: Finished 1.31200e+00
[ Info: 3: Finished 1.32000e+00
[ Info: 2: Finished 1.32500e+00
[ Info: Finished 1.32600e+00
[ Info: nthreads = 5
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:default, :default, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (5, 0)
[ Info: Chunk 3, thread 4, default: Spawned 0.00000e+00
[ Info: Chunk 4, thread 1, default: Spawned 0.00000e+00
[ Info: Chunk 2, thread 3, default: Spawned 0.00000e+00
[ Info: Chunk 1, thread 2, default: Spawned 0.00000e+00
[ Info: 4: Finished 1.28900e+00
[ Info: 2: Finished 1.28900e+00
[ Info: 1: Finished 1.28900e+00
[ Info: 3: Finished 1.29000e+00
[ Info: Finished 1.29000e+00
extrema(ts) = (1.29278e+00, 1.32864e+00)
(1.29278e+00, 1.32864e+00)

1. When running julia interactively, no interactive thread is reported. However, maxthreadid = 6 is printed, when I asked for 5 threads.
2. When not running interactively, the number of threads and the pool assignations are identical.

Now I start julia with two thread pools configured explicitly:

>>>>>>>>>>>>> INTERACTIVE <<<<<<<<<<<<<<<<<<<
$ julia -t4,1

julia> include(raw"C:\Users\pkonl\Documents\00WIP\FinEtoolsHeatDiff.jl\examples\steady_state\3-d\mwe_tasks.jl")
[ Info: nthreads = 4
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:interactive, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (4, 1)
[ Info: Chunk 1, thread 3, default: Spawned 5.90000e-02
[ Info: Chunk 4, thread 2, default: Spawned 5.15000e-01
[ Info: Chunk 3, thread 5, default: Spawned 5.19000e-01
[ Info: Chunk 2, thread 4, default: Spawned 6.30000e-02
[ Info: 1: Finished 1.79800e+00
[ Info: 4: Finished 1.80000e+00
[ Info: 2: Finished 1.80200e+00
[ Info: 3: Finished 1.80600e+00
[ Info: Finished 1.80700e+00
[ Info: nthreads = 4
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:interactive, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (4, 1)
[ Info: Chunk 2, thread 2, default: Spawned 0.00000e+00
[ Info: Chunk 3, thread 4, default: Spawned 9.99928e-04
[ Info: Chunk 1, thread 5, default: Spawned 0.00000e+00
[ Info: Chunk 4, thread 3, default: Spawned 9.99928e-04
[ Info: 3: Finished 1.28700e+00
[ Info: 4: Finished 1.28700e+00
[ Info: 1: Finished 1.28800e+00
[ Info: 2: Finished 1.28900e+00
[ Info: Finished 1.28900e+00
[ Info: nthreads = 4
[ Info: maxthreadid = 6
[ Info: nthreadpools = 2
[ Info: threadpool.(1:Threads.nthreads()) = [:interactive, :default, :default, :default]
[ Info: Threads.threadpoolsize.((:default, :interactive)) = (4, 1)
[ Info: Chunk 4, thread 5, default: Spawned 0.00000e+00
[ Info: Chunk 1, thread 2, default: Spawned 0.00000e+00
[ Info: Chunk 3, thread 4, default: Spawned 0.00000e+00
[ Info: Chunk 2, thread 3, default: Spawned 0.00000e+00
[ Info: 3: Finished 1.28400e+00
[ Info: 2: Finished 1.28700e+00
[ Info: 4: Finished 1.28800e+00
[ Info: 1: Finished 1.28900e+00
[ Info: Finished 1.28900e+00
extrema(ts) = (1.29101e+00, 1.29140e+00)
(1.29101e+00, 1.29140e+00)

1. The number of threads is reported as 4. Is it because only the default pool threads are counted?
2. When listing the thread pool assignations, one of them is reported as interactive. Why?
3. The tasks are however all started from the default pool.

[samtkaplan]

@PetrKryslUCSD, I'll try to answer your three questions, but I'm guessing we should move this type of discussion to the discourse thread.

1. Threads.nthreads() is equivalent to Threads.nthreads(:default), so it only reports the count for the default thread pool. UseThreads.nthreads(:interactive) to see the number of threads in the interactive thread pool.
2. You are reporting the thread-pools associated with individual threads. You can use Threads.threadpooltids(:default) and Threads.threadpooltids(:interactive) to see the which thread ids are associated with each pool.
3. Threads.@spawn ... is equivalent to Threads.@spawn :default .... Use Threads.@spawn :interactive ... to run a task on the interactive thread pool.

[carstenbauer]

Doc fix: #53388

Regarding the question of why the main thread can be in either the default or interactive threadpool, I'll open a separate issue to discuss this, because the present issue is a bit convoluted. (done: #53389)