Add registered_domain processor similar to Filebeat processor

[leehinman]

It would be nice to have a registered_domain processor like the filebeat one. This would be useful so we can have ingest pipelines that are completely in ES that can fill out the ECS fields like dns.question.top_level_domain.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[jamiehynds]

Discussed with @jakelandis. @jamiehynds to provide ECS fields to map to.

[webmat]

In ECS, fields that contain a DNS domain name can appear in many places:

• url.domain generally contains an FQDN, but can also contain an IP address
• dns.question.name contains a domain in CNAME lookups, but can can contain other things, depending on the DNS query type
• [source|destination|client|server].domain can contain either FQDNs or Active Directory domain names.

Each of these field can potentially contain stuff that doesn't look like a FQDN, so leniency is important. Or at least the code path for invalid values should be as fast :-)

The convention we've followed has always been to have the domain breakdown fields as siblings to each of the above. The breakdown fields are .registered_domain, .top_level_domain and subdomain. You can check out each of their descriptions on the ECS dns fields page.

But by example, when breaking down "www.google.co.uk":

• registered_domain: google.co.uk
• top_level_domain: co.uk (so it's the "effective" TLD)
• subdomain: www

So I think this processor's output target would could default to outputting the breakdown fields as siblings to the field from which the value is being read. E.g. dns.question.name => dns.question.registered_domain, dns.question.top_level_domain, dns.question.subdomain

[mbudge]

Hi,

We are implementing indicator match in the SIEM. One issue we have is doing TLD extract in non-beats fieds i.e. not packetbeat.

Proxy data is indexed in elastic directly through logstash, and although there is a Logstash TLD extract filter it is not maintained/documented.

The problem is if an indicator has a sub-domain in the logs but not the threat intel feed, there's a high risk it will be missed.

It would help if we could do TLD extract in an ingest processor.

The rules for doing TLD extract with the public suffix list can be found here
https://publicsuffix.org/list/