[Monitoring] HTTP Exporter should support proxy settings / path

[elasticmachine]

Original comment by @pickypg:

Currently, the HTTP Exporter very loosely supports proxy paths (subpaths after the URL), but it does not support a proxy host or port. With LINK REDACTED, proxy.base_path will exist in order to continue to support a mechanism (base paths) that has silently existed since 2.0.

xpack.monitoring.exporters._http:
  type: http
  host: "localhost:9200"
  proxy:
    host: someproxy
    port: 8200
    base_path: /some/path

We should provide additional settings to support proxies.

[elasticmachine]

Original comment by @pickypg:

This has existed since 5.0.

[elasticmachine]

Original comment by @tlrx:

I'm reopening this one because I don't think that monitoring exporters supports proxy.host and proxy.port settings, and now we are offering free Elastic Stack Monitoring Service customers are asking where to configure proxies (see LINK REDACTED and others).

This question has been in the backlog of many BugPools but it has never been discussed. I'm not sure if there's any security reason in not implementing this, and I think it should be implemented soon.

[elasticmachine]

Original comment by @GlenRSmith:

Subscribe

[elasticmachine]

Original comment by @gmoskovicz:

CC @pickypg any plans on adding this?

[elasticmachine]

Original comment by @gmoskovicz:

Starting from 5.6.7 it seems that #27829 is preventing you from using a proxy for Monitoring as the JVM parameters don't work any longer.

CC @jaymode @pickypg

[elasticmachine]

Original comment by @pickypg:

This question has been in the backlog of many BugPools but it has never been discussed. I'm not sure if there's any security reason in not implementing this, and I think it should be implemented soon.

I'm not aware of any true issue. We could add the settings and manipulate the HttpAsyncClient as needed.

[elasticmachine]

Original comment by @jaymode:

Support for this was unintentionally added with LINK REDACTED which happened to break other things so it was rolled back in #27829

[elasticmachine]

Original comment by @gmoskovicz:

Indeed, i seem to see this working in 5.6.1 .

I think that we could add a monitoring specific setting to build the HttpAsyncClient for this? I see previous comments.

[ckloote]

I would also be interested in having this as a feature. We run several good-sized Elastic clusters on a private network and would like to make use of the cloud monitoring service.

Thanks.

[JalehD]

@ycombinator @elasticmachine is this in the roadmap?

[ycombinator]

@JalehD Thanks for the ping. We discussed this in our weekly team meeting this morning. The thinking is that rather than implement a proxy setting in ES, we ask users to use Metricbeat for monitoring instead, which is the long-term plan anyway.

However, before we make this recommendation:

1. The setting in Metricbeat for configuring a proxy for the Elasticsearch output is https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html#_literal_proxy_url_literal. Will this get you what you need or do you need something more?

2. It would be good to know if users are looking to monitor ES or Kibana or LS or Beats? The reason I ask is we currently support monitoring of ES and Kibana using Metricbeat and are working on building support for LS and Beats next.

[JalehD]

@ycombinator this issue affects the usability of this service for those who have servers that must connect to internet through proxy and this use-case I'm afraid won't be addressed by your above suggestions as it requires the ability to configure proxy settings in Monitoring Exporter configuration.