Logstash health_report API status not effected by pipeline errors

[DanielBrown2023]

Logstash information:

1. Logstash version (e.g. bin/logstash --version)
   9.0.2

2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
   zip archive

3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
   command line

Plugins installed: (bin/logstash-plugin list --verbose)
None

JVM (e.g. java -version):
Bundled jdk

OS version (uname -a if on a Unix-like system):
Mac Darwin

Description of the problem including expected versus actual behavior:

actual behavior: health_report API shows status of GREED, while a input plugin is reporting errors.

expected behavior: health_report API would capture errors related to plugin failures.

Senario:

Logstash is started with a tcp input listening to port 1025, but another process is currently bound to that process preventing logstash from binding to it.

input {
  tcp {
    port => 1025
  }
}

Logs excerpt of the error

[2025-06-10T12:07:54,025][ERROR][logstash.javapipeline    ][main][9664765ab09fa8ef6961b8cdbbe1ee6106bd5ae1f2b0190868c50d378e9a4760] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Tcp port=>50000, id=>"9664765ab09fa8ef6961b8cdbbe1ee6106bd5ae1f2b0190868c50d378e9a4760", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_ad5dfa8c-e495-45a3-aed6-c361a26fd407", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enabled=>false, ssl_client_authentication=>"required", ssl_verification_mode=>"full", ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
  Error: event executor terminated
  Exception: Java::JavaUtilConcurrent::RejectedExecutionException
  Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:482)
io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89)
io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83)
io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)

health_report API response

{
  "host": "Daniels-MacBook-Pro.local",
  "version": "9.0.2",
  "http_address": "127.0.0.1:9601",
  "id": "ed127875-ea31-4646-8668-57d2f1040f2e",
  "name": "Daniels-MacBook-Pro.local",
  "ephemeral_id": "ec2922f8-fcbe-4993-9317-f4ad0a3bdcdc",
  "snapshot": false,
  "status": "green",
  "symptom": "1 indicator is healthy (`pipelines`)",
  "indicators": {
    "pipelines": {
      "status": "green",
      "symptom": "1 indicator is healthy (`main`)",
      "indicators": {
        "main": {
          "status": "green",
          "symptom": "The pipeline is healthy",
          "details": {
            "status": {
              "state": "RUNNING"
            },
            "flow": {
              "worker_utilization": {
                "last_1_minute": 0.0005793,
                "lifetime": 0.0008709,
                "last_15_minutes": 0.0006902,
                "last_1_hour": 0.0008608,
                "current": 0.0008516,
                "last_5_minutes": 0.0006855
              }
            }
          }
        }
      }
    }
  }
}

[yaauie]

I acknowledge that it would be helpful to have this situation trigger a degredation.

From the perspective of the agent, the pipeline is healthy (even though it obviously isn't); the pipeline is running. A relevant detail that isn't visible to the agent is that the pipeline has an input plugin that is crashing and relaunching.

In this case, the TCP input is not binding to the port during the #register phase (which would be escalated by the pipeline as an error), and instead is doing so in the #run phase (which is treated as an internal detail of the pipeline).

I believe that our answer needs to be two-fold:

1. define the boundary of the input plugin with regard to where errors are interpreted as transient vs fatal. In the pipeline's current implementation I believe this boundary is effectively #register => fatal and #run => transient, but we will need to audit the default plugins to ensure that they are invoked on the correct side of that line.
2. provide a way for the pipeline to register a possibly-time-limited degraded status that returns a handle by which that specific degradation can be removed, and wire up the JavaPipeline#inputworker(plugin) to mark when an input crashes and clear when the input successfully restarts.