"dispatch dropped without returning error" panic

[alecmocatta]

I'm getting a panic in hyper while making async requests with rusoto on multiple threads. It occurs intermittently, though with increasing frequency as I increase the number of threads I'm using to make requests.

I create futures on multiple threads like this:

Box::pin(async move {
    let (start, end) = (offset, offset + u64::try_from(buf.len()).unwrap() - 1);
    // this line is amadeus-aws/src/file.rs:221 in the backtrace
    let res =
        futures::compat::Compat01As03::new(self.client.get_object(GetObjectRequest {
            bucket: self.bucket.clone(),
            key: self.key.clone(),
            range: Some(format!("bytes={}-{}", start, end)),
            ..GetObjectRequest::default()
        }))
        .await
        .unwrap();
    let len: u64 = buf.len().try_into().unwrap();
    let mut cursor = io::Cursor::new(buf);
    let mut read = res.body.unwrap().into_async_read();
    while len - cursor.position() > 0 {
        let _: usize =
            futures::compat::Compat01As03::new(tokio::prelude::future::poll_fn(|| {
                read.read_buf(&mut cursor)
            }))
            .await
            .unwrap();
    }
    Ok(())
})

And block on them on the same thread they are created on like so:

// this line is amadeus-aws/src/file.rs:17 in the backtrace
tokio::runtime::current_thread::Runtime::new()
    .unwrap()
    .block_on(futures::compat::Compat::new(
        Box::pin(future).map(Ok::<_, ()>),
    ))
    .unwrap()

Prior to this I also block on rusoto futures using RusotoFuture::sync(). Other than that there is no invocation of tokio/hyper/rusoto.

It does not panic if I wrap the blocking in a mutex, i.e. limit the resolving of futures to one thread at a time.

The panic is at: https://github.com/hyperium/hyper/blob/0.12.x/src/client/conn.rs#L259

I'll look into this further tomorrow but any pointers or suggestions would be much appreciated!

   thread '<unnamed>' panicked at 'dispatch dropped without returning error', ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/hyper-0.12.34/src/client/conn.rs:259:35
      0: backtrace::backtrace::libunwind::trace
                at /Users/vsts/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/backtrace-0.3.35/src/backtrace/libunwind.rs:88
      1: backtrace::backtrace::trace_unsynchronized
                at /Users/vsts/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/backtrace-0.3.35/src/backtrace/mod.rs:66
      2: std::sys_common::backtrace::_print
                at src/libstd/sys_common/backtrace.rs:47
      3: std::sys_common::backtrace::print
                at src/libstd/sys_common/backtrace.rs:36
      4: std::panicking::default_hook::{{closure}}
                at src/libstd/panicking.rs:200
      5: std::panicking::default_hook
                at src/libstd/panicking.rs:214
      6: std::panicking::rust_panic_with_hook
                at src/libstd/panicking.rs:477
      7: std::panicking::begin_panic
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/panicking.rs:411
      8: hyper::client::conn::SendRequest<B>::send_request_retryable::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/rusoto_core-0.40.0/<::std::macros::panic macros>:3
      9: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/then.rs:33
     10: futures::future::chain::Chain<A,B,C>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/chain.rs:39
     11: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/then.rs:32
     12: <futures::future::either::Either<A,B> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/either.rs:35
     13: <futures::future::either::Either<A,B> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/either.rs:35
     14: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/map_err.rs:30
     15: <futures::future::map::Map<A,F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/map.rs:30
     16: futures::future::chain::Chain<A,B,C>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/chain.rs:26
     17: <futures::future::and_then::AndThen<A,B,F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/and_then.rs:32
     18: <futures::future::either::Either<A,B> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/either.rs:36
     19: <futures::future::either::Either<A,B> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/either.rs:36
     20: futures::future::chain::Chain<A,B,C>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/chain.rs:32
     21: <futures::future::and_then::AndThen<A,B,F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/and_then.rs:32
     22: hyper::client::Client<C,B>::retryably_send_request::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/hyper-0.12.34/src/client/mod.rs:266
     23: <futures::future::poll_fn::PollFn<F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/poll_fn.rs:43
     24: <alloc::boxed::Box<F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/mod.rs:113
     25: <hyper::client::ResponseFuture as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/hyper-0.12.34/src/client/mod.rs:613
     26: <rusoto_core::request::HttpClientFuture as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/rusoto_core-0.40.0/src/request.rs:238
     27: <rusoto_core::client::SignAndDispatchFuture<P,D> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/rusoto_core-0.40.0/src/client.rs:184
     28: <alloc::boxed::Box<F> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/future/mod.rs:113
     29: <rusoto_core::future::RusotoFuture<T,E> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/rusoto_core-0.40.0/src/future.rs:244
     30: core::ops::function::FnOnce::call_once
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libcore/ops/function.rs:227
     31: futures::task_impl::Spawn<T>::enter::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
     32: futures::task_impl::std::set
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/std/mod.rs:83
     33: futures::task_impl::Spawn<T>::enter
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
     34: futures::task_impl::Spawn<T>::poll_fn_notify
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:291
     35: futures_util::compat::compat01as03::Compat01As03<T>::in_notify
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/compat/compat01as03.rs:42
     36: <futures_util::compat::compat01as03::Compat01As03<Fut> as core::future::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/compat/compat01as03.rs:165
     37: std::future::poll_with_tls_context::{{closure}}
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:120
     38: std::future::get_task_context
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:110
     39: std::future::poll_with_tls_context
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:120
     40: <amadeus_aws::file::S3Page as amadeus_core::file::Page>::read::{{closure}}
                at amadeus-aws/src/file.rs:221
     41: <std::future::GenFuture<T> as core::future::future::Future>::poll::{{closure}}
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:42
     42: std::future::set_task_context
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:78
     43: <std::future::GenFuture<T> as core::future::future::Future>::poll
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/future.rs:42
     44: <core::pin::Pin<P> as core::future::future::Future>::poll
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libcore/future/future.rs:119
     45: <futures_util::future::map::Map<Fut,F> as core::future::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/future/map.rs:40
     46: <core::pin::Pin<P> as core::future::future::Future>::poll
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libcore/future/future.rs:119
     47: <futures_util::future::map::Map<Fut,F> as core::future::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/future/map.rs:40
     48: <F as futures_core::future::TryFuture>::try_poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-core-preview-0.3.0-alpha.18/src/future/mod.rs:86
     49: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/compat/compat03as01.rs:122
     50: futures_util::compat::compat03as01::with_context
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/compat/compat03as01.rs:224
     51: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.18/src/compat/compat03as01.rs:122
     52: futures::task_impl::Spawn<T>::poll_future_notify::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
     53: futures::task_impl::Spawn<T>::enter::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
     54: futures::task_impl::std::set
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/std/mod.rs:83
     55: futures::task_impl::Spawn<T>::enter
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
     56: futures::task_impl::Spawn<T>::poll_fn_notify
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:291
     57: futures::task_impl::Spawn<T>::poll_future_notify
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
     58: tokio_current_thread::Entered<P>::block_on::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494
     59: tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
     60: tokio_current_thread::CurrentRunner::set_spawn
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
     61: tokio_current_thread::Borrow<U>::enter::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
     62: std::thread::local::LocalKey<T>::try_with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:262
     63: std::thread::local::LocalKey<T>::with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:239
     64: tokio_current_thread::Borrow<U>::enter
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
     65: tokio_current_thread::Entered<P>::block_on
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491
     66: tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:200
     67: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:241
     68: tokio_executor::global::with_default::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:209
     69: std::thread::local::LocalKey<T>::try_with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:262
     70: std::thread::local::LocalKey<T>::with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:239
     71: tokio_executor::global::with_default
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:178
     72: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:239
     73: tokio_timer::timer::handle::with_default::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:101
     74: std::thread::local::LocalKey<T>::try_with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:262
     75: std::thread::local::LocalKey<T>::with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:239
     76: tokio_timer::timer::handle::with_default
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:84
     77: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:232
     78: tokio_timer::clock::clock::with_default::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:137
     79: std::thread::local::LocalKey<T>::try_with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:262
     80: std::thread::local::LocalKey<T>::with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:239
     81: tokio_timer::clock::clock::with_default
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:117
     82: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:231
     83: tokio_reactor::with_default::{{closure}}
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:237
     84: std::thread::local::LocalKey<T>::try_with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:262
     85: std::thread::local::LocalKey<T>::with
                at /rustc/fba38ac27e2ade309f4c2504a6d6cd3556972a28/src/libstd/thread/local.rs:239
     86: tokio_reactor::with_default
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:217
     87: tokio::runtime::current_thread::runtime::Runtime::enter
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:230
     88: tokio::runtime::current_thread::runtime::Runtime::block_on
                at ~/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:198
     89: amadeus_aws::file::block_on
                at ./amadeus-aws/src/file.rs:17
   ...

[alecmocatta]

I attribute this to "creating multiple tokio::runtime::current_thread::Runtimes".

I tried caching the Runtimes in a thread_local so there's at most 1 per thread, but this gave a panic inside tokio. In the end I gave up and used tokio::runtime::Runtime.