[agents] Read k8s data from environment variables

[roncohen]

It would be great if agents could read kubernetes data from environment variables and add it automatically to metadata sent to APM Server. This would work both for kubernetes setups where APM Server runs self-hosted and in the future where APM Server could be hosted.

I'll file a separate issue for the APM Server work.

Users set the environment variables through the DownwardAPI.

Here's a table mapping the environment variable to the metadata event field:

Environment variable	Field name
KUBERNETES_NODE_NAME	system.kubernetes.node.name
KUBERNETES_POD_NAME	system.kubernetes.pod.name
KUBERNETES_NAMESPACE	system.kubernetes.namespace
KUBERNETES_POD_UID	system.kubernetes.pod.uid

Elasticsearch documents will use kubernetes.node.name (and not system.kubernetes.node.name) etc., so it aligns with the fields generated by filebeat, except for a few:

• container.* fields which we don't seem to have access to unfortunately.
• annotations: off by default in filebeat because people use them to store all sorts of large objects
• labels: could be very useful down the line, but lets start without as an MVP

Here's how a user would add the environment variables to their kubernetes pod spec:

         - name: KUBERNETES_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: KUBERNETES_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: KUBERNETES_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: KUBERNETES_POD_UID
            valueFrom:
              fieldRef:
                fieldPath: metadata.uid

@elastic/apm-agent-devs Please link your issue or PR and then mark the checkbox when the implementation is merged:

• - Java - Read k8s data from environment variables apm-agent-java#344
• - Python Add kubernetes metadata to system data apm-agent-python#347
• - Ruby - Read k8s data from environment variables apm-agent-ruby#252
• - Go - Add k8s pod metadata apm-agent-go#329
• - dotNet - Report kubernetes system metadata apm-agent-dotnet#181
• - Node.js - feat: add Kubernetes data to metadata apm-nodejs-http-client#34
• - Documentation - Docs: add kubernetes metadata apm-server#1650
• - APM Server Allow APM Server to receive k8s/container information from agents apm-server#1621

Potential additional fields that we should not do now

Environment variable	Field name
KUBERNETES_POD_IP	system.kubernetes.pod.ip
KUBERNETES_NODE_IP	system.kubernetes.node.ip
KUBERNETES_LABELS_*	system.kubernetes.labels.*

          - name: KUBERNETES_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: KUBERNETES_NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: KUBERNETES_LABELS_APP
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['app']

[roncohen]

Updated to reflect that data should be added to metadata under system.

[watson]

I assume we can just publish this in the agents without coordinating with a specific release of the APM Server?

[ruflin]

@roncohen I think the k8s fields are without a system.* prefix.
@exekias For awareness.

[roncohen]

@ruflin this is for agent devs. The intake api uses system. I'll make it more clear

[beniwohli]

Where should we document how the user has to set the environment variables? In a central location or in each of the agent docs?

[roncohen]

@bmorelli25 probably makes sense to describe the process for setting up the k8s environment variables somewhere centrally instead of in each agent docs. WDYT?

[watson]

I was thinking we probably should still link to this central page from each agent. That got me thinking if asciidoc might even allow us to "embed" this central docs into the actual agent docs. But maybe this is too complicated or doesn't make sense. Anyway, just a thought 😃

[bmorelli25]

@watson, your idea would work. We could create an asciidoc file in the APM Overview directory, then include:: it in each agent's documentation. Wouldn't be difficult to implement or maintain, I just worry that it might overly complicate things for users. Having the same information six times scares me a bit. If the documentation is identical, I prefer the centralized docs idea (like in APM Overview) with all agents linking to it.

[axw]

@roncohen in the description you've got KUBERNETES_POD_NAMESPACE -> system.kubernetes.namespace

I think that should just be KUBERNETES_NAMESPACE?

EDIT: I guess not, given that https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables also uses "MY_POD_NAMESPACE". I suppose that makes sense since it's only relevant to the pod, and not the node. I was kinda thrown by the fact that the JSON path doesn't have "pod" in it.