[agents] Read docker container id from filesystem

[roncohen]

Agents should try to read the current container ID from the local filesystem. Elasticsearch already does this.

There's previously been discussion around issues that could arise because the agent could be blocked while fetching this information. I'm not too concerned about blocking the agent startup for a little bit while we check if the file exists and read/parse it. It should only take a few milliseconds. If we start supporting AWS Lambda and similar we could skip the check outright based on an environment variable that is automatically set by the provider.

There are some complications involved in parsing the file so it would be nice if one of the agents buildt a reference implementation that others can take model their own implementation on.

Pleas add the field to metadata as system.container.id.

The field will end up in Elasticsearch as container.id: https://github.com/elastic/ecs/blob/master/schemas/container.yml#L19-L23

@elastic/apm-agent-devs Please link your issue or PR and then mark the checkbox when the implementation is merged:

• - Java - Read docker container id from filesystem apm-agent-java#325
• - Python Add docker container metadata to system data apm-agent-python#346
• - Ruby - Read docker container id from filesystem apm-agent-ruby#253
• - Go - Add container ID to metadata apm-agent-go#328
• - dotNet - Send container.id in system. apm-agent-dotnet#238
• - Node.js - Add Docker container ID to metadata apm-agent-nodejs#693
• - APM Server - Allow APM Server to receive k8s/container information from agents apm-server#1621

[axw]

Please only add the field to transactions and metrics.

Shouldn't this be going in metadata, and not in specific events?

[axw]

Also

The field should be container.id: https://github.com/elastic/ecs/blob/master/schemas/container.yml#L19-L23

That is relevant to the server. Where should we put it in the events we send to the server?

In metadata we currently have system, service, process, and user. We could add this as either container.id or system.container.id, seeing as in the case of containerization system relates to the container. Preferences?

[watson]

I vote for system.container.id, but inventing a new root property called container might also work 👍

[axw]

system.container.id is my preference also. My PR is here, in case anyone wants to copy: elastic/apm-agent-go#330. I'll hold off on merging until we come to a decision on the schema change.

[roncohen]

you're right. I'll update the description. Thanks!

[watson]

As this is a blocking operation on boot, should we have a config option to disable it?

[axw]

As this is a blocking operation on boot, should we have a config option to disable it?

What are the conditions under which reading that path will block? Can you please point to some literature?

[beniwohli]

Does anybody have an exhaustive list of possible formatting that can occur in /proc/self/cgroup? Googling a bit, it doesn't seem to be a super stable file format.

[watson]

@axw It's in the issue description:

There's previously been discussion around issues that could arise because the agent could be blocked while fetching this information. I'm not too concerned about blocking the agent startup for a little bit while we check if the file exists and read/parse it

[beniwohli]

@watson /proc is a virtual file system and doesn't involve any disk I/O. So while reading a file from it is technically still a blocking operation, I don't think we have to expect it to be blocking startup.