Data Source Categorization Fields

[jamiehynds]

Summary

Elastic currently supports ingestion of data from 180+ sources, and growing. However, we do not have a coherent way to categorise these sources. This has resulted in a disconnect in how we categorize these sources from the Elastic website, in-product experiences and ECS.

Motivation:

Categorization fields in ECS can govern how we categorize these data source, but only a limited set of event.category values are supported by the schema today. The new dataset fields should also support these values, possibly under dataset.type. Expanding the values we support, allows us to align the user experience from ECS, Ingest Manager and the Elastic Website (elastic.co/integrations). Some additional context here: #845 (comment)

Detailed Design:
Here are some of the proposed values that @exekias and I propse:

• apm
• application
• audit
• cloud
• collaboration
• Config Management
• containers
• CRM
• email
• firewall
• Operating System
• productivity
• queue/message queue
• security
• storage
• threat
• ticketing
• vulnerability

[webmat]

Love the idea, thanks @jamiehynds!

Our current categorization fields are aimed at capturing the essence of what's in a single event. A given source typically produce more than one category of such events. E.g. A firewall can often emit events around network flows, authentications, etc.

However I think having a straightforward way to categorize sources will be helpful as well (e.g. this is a firewall).

[jamiehynds]

Thanks @webmat! @exekias and I will work together on an RFC to move forward.

[jamiehynds]

Closing the issue. Discussion can continue via the RFC: #958

[jamiehynds]

@approksiu issue discussing data source types.