Specifying types in bulk requests is deprecated

[fdartayre]

Version: logstash-output-elasticsearch 10.2.3 (shipping with Logstash 7.5.2)

Starting from Elasticsearch 7.0, types should also no longer appear in the body of bulk requests.

Including a type causes a deprecation warning to get logged in Elasticsearch:

[2020-01-31T10:58:52,319][WARN ][o.e.d.a.b.BulkRequestParser] [hostname] [types removal] Specifying types in bulk requests is deprecated.

The plugin shouldn't use types anymore.

[andsel]

With commit c93abd1 present in version 10.2.2 type was removed but with 39a4db9 was added because the _monitoring/bulk operation required it to continue work. With the LS 7.7 the end point should be switched and then removed

[rolffokkens]

Currently at LS 7.10, end the ES (also 7.10) logs are flouding my filesystem. No idea what the previous comment means. The flouding ("[types removal] Specifying types in bulk requests is deprecated.") started after upgrading ES from 7.9 to 7.10.

[barosch78]

Currently at LS 7.10, end the ES (also 7.10) logs are flouding my filesystem. No idea what the previous comment means. The flouding ("[types removal] Specifying types in bulk requests is deprecated.") started after upgrading ES from 7.9 to 7.10.

I have the same problem after upgrading to 7.10

[DrBautze]

Same problem here. Flooding can be stopped by changing log level of deprecation warnings.

From: https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html

You can disable it in the config/log4j2.properties file by setting the deprecation log level to error like this:

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = error

This is merely a workaround and I would appreciate a fix.

[MakoWish]

Any updates on this issue?

[andsel]

Could you share some information about this problem please, which logstash-output-elasticseach configuration are you using?Did you remove the type from your config definition when pushing data to Elasticsearch?

[MakoWish]

Hi @andsel,

We are sending Beats data through Logstash for some additional data massaging. The entire stack is 7.10.2. That includes Beats, Logstash, Elasticsearch, Kibana...

Beats output:

output.logstash:
  hosts: [
    "LS1.Contoso.com:5044",
    "LS2.Contoso.com:5044",
    "LS3.Contoso.com:5044",
    "LS4.Contoso.com:5044"
  ]
  ssl.certificate: "${path.config}\\certs\\client.crt"
  ssl.key: "${path.config}\\certs\\client.key"
  ssl.key_passphrase: "${SECRET_PASS}"

Logstash output:

output {
  # Output for beats modules with a pipeline declared
  if [@metadata][pipeline] and "beat" in [agent][type] {
    elasticsearch {
      hosts => [
        "https://ES1.Contoso.com:9200",
        "https://ES2.Contoso.com:9200",
        "https://ES3.Contoso.com:9200",
        "https://ES4.Contoso.com:9200",
        "https://ES5.Contoso.com:9200"
      ]
      manage_template => false
      user => "somebody"
      password => "${SUPER_DUPER_SECRET}"
      index => "%{[@metadata][beat]}"
      pipeline => "%{[@metadata][pipeline]}" 
    }
  }

  # Output for all remaining beats modules (no pipeline)
  else if "beat" in [agent][type] {
    elasticsearch {
      hosts => [
        "https://ES1.Contoso.com:9200",
        "https://ES2.Contoso.com:9200",
        "https://ES3.Contoso.com:9200",
        "https://ES4.Contoso.com:9200",
        "https://ES5.Contoso.com:9200"
      ]
      manage_template => false
      user => "somebody"
      password => "${SUPER_DUPER_SECRET}"
      index => "%{[@metadata][beat]}"
    }
  }
}

Nearly all our data is from Beats modules, using the included Beats index templates and pipelines. None of our non-Beats sources have a "_type" field explicitly added. It seems (I don't know where it is being added to prove) either Beats or Logstash are adding a field:value of "_type" : "_doc" to every event ingested. Basically, I am not adding a "_type" field anywhere, so it must be part of the stack itself?

Would throwing something in the Logstash pipeline like this do the trick?

mutate {
    remove_field => ["_type"]
}

Just to be 100% sure I wasn't adding a "_type" field anywhere, I ran sudo cat /etc/logstash/conf.d/*.conf | grep -i "_type". The only things found were fields from grok match patterns (ie. EVENT_TYPE = %{WORD:[event][type]} matched).

Here is just few of the literally millions of these events we receive per day. Even the event saying not to use the "_type" field is for some reason still using the "_type" field, LOL! Again, I am not adding this myself.

[MakoWish]

UPDATE: I tried removing the "_type" field in my pipeline as noted above, so that should rule out the Beats agent adding it, but it is still in every event.

[andsel]

@MakoWish are you indexing Elasticsearch's deprecation logs in Elasticsearch itself?
it should be the index used by upgrade UI that contains all the log lines from deprecation logger.

[andsel]

It's assigned by logstash-output-elasticsearh plugins at:

logstash-output-elasticsearch/lib/logstash/outputs/elasticsearch.rb

Line 328 in 3769e2d

params[:_type] = get_event_type(event) if use_event_type?(nil)

with value:

logstash-output-elasticsearch/lib/logstash/outputs/elasticsearch.rb

Line 388 in 3769e2d

DEFAULT_EVENT_TYPE_ES7="_doc".freeze

[MakoWish]

@andsel,

Thank you for the response! Will that be disabled automatically with v8? Or is there a way to disable it now? I would like to stop the literally millions of messages we receive daily.

[andsel]

A reproducer for this:

• download Elasticsearch 7.10.0 tar.gz, unpack and run Elasticsearch bin/elasticsearch
• download Logstasn 7.10.0, unpack it
• create a pipeline config (name it test_pipeline.conf) with content:

input {
	generator {
		message => "{\"name\": \"John Doe\"}"
		count => 20000
	}
}

output {
	elasticsearch {
		index => "test_bulk_index"
	}
}

• run bin/logstash -f test_pipeline.conf
• check the logs/elasticserach_deprecation.log increasing

[andsel]

Logstash output Elasticsearch plugin uses the type when connect to a cluster of version < 8.0. Working on PR #994 to avoid it's usage also when connecting to Elasticsearch 7 and user doesn't specify type setting in plugin configuration.

As a fix on Elasticsearch side, instead of changing the level of deprecation logger:

• search for Deprecation - old style pattern section in config/log4j2.properties in Elasticsearch
• append this config appender.deprecation_rolling_old.filter.rate_limit.type = RateLimitingFilter