Processor actor terminated abruptly on HTTPs connections (akka 2.4.12, akka-http 2.4.11)

[samidalouche]

I posted on akka-user, but since I believe it might be a bug, I am opening an issue for that.

The problem seems to be that when performing HTTPS queries using a connection pool, shutting down the actor system raises the following error:

ERROR akka.actor.ActorSystemImpl - Outgoing request stream error
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://default/user/StreamSupervisor-1/flow-0-0-unknown-operation#-901476425]] terminated abruptly

An example of simple code that reproduces the problem (replace the https prefix by http in the uri field to make the problem disappear)

object HttpsWithConnectionPoolIssue extends App {

  implicit val system = ActorSystem()
  implicit val materializer = ActorMaterializer()
  implicit val timeout = Timeout(10.seconds)
  import system.dispatcher

  val request = HttpRequest(
    uri = "https://httpstatuses.com/200"
  )

  val future = Http(system)
    .singleRequest(request)
    .map(_.discardEntityBytes())

  val systemTerminatedFuture = future.flatMap { _ =>
    Http().shutdownAllConnectionPools().flatMap { _ =>
      materializer.shutdown()
      system.terminate()
    }
  }

  Await.result(systemTerminatedFuture, Duration.Inf)
}

An example of code that does not exhibit the problem (no connection pool)

object HTTPsConnectionNotExhibitingProblem extends App {

  implicit val system = ActorSystem()
  implicit val materializer = ActorMaterializer()
  implicit val timeout = Timeout(10.seconds)
  import system.dispatcher

  val request = HttpRequest(
    uri = "/200"
  )

  val flow = Http(system).outgoingConnectionHttps("httpstatuses.com")

  val source = Source.single(request)
    .via(flow)
    .map(_.discardEntityBytes())

  val sourceProcessed = source.runWith(Sink.ignore)

  val systemTerminatedFuture = sourceProcessed.flatMap { _ =>
    Http().shutdownAllConnectionPools().flatMap { _ =>
      materializer.shutdown()
      system.terminate()
    }
  }

  Await.result(systemTerminatedFuture, Duration.Inf)
}

[jrudolph]

This error message is probably harmless. The reason it occurs is that when an ActorSystem is shutdown it stops all its actors. The streaming infrastructure relies on those actors and then complains that something has shutdown the actors. This happens in particular with the connection pool actors as these streams need to be kept alive for future requests.

Still, it's unfortunate, that these expected errors are logged verbosely in the console (we are also seeing them in tests). Here's a related ticket from akka to improve the situation: akka/akka-core#18747

[jonas]

It's also somewhat related with #450.

[arturaz]

I have also stumbled upon this. Repro at https://github.com/arturaz/akka-http-https-bug-repro

[BusyByte]

We are also seeing this issue. It's harmless as the code executes but really confusing to the users of our CLI which uses akka-http as they think there's been an error when nothing is wrong.

[BusyByte]

We are using singleRequest and shutdown like:

Await.result(httpExt.shutdownAllConnectionPools(), shutDownWaitTime)
Await.result(actorSystem.terminate(), shutDownWaitTime)

[jrudolph]

I created #907 to track further improvements in the shutdown process.

Closing here.