[k8sobjectsreceiver] Degrade gracefully, when watched CRDs are not found

[krisztianfekete]

Component(s)

receiver/k8sobjects

What happened?

Description

Today, if you specify a list of CRDs to be watched, the collector won't start if any of them are missing and stay in CrashLoopBackOff.

Steps to Reproduce

Apply the following receiver and use it in a pipeline without having telemetries.telemetry.istio.io installed on your cluster.

receivers:
    k8sobjects:
      objects:
        - name: telemetries
          mode: watch
          group: telemetry.istio.io/v1alpha1

Expected Result

The collector should start, and the receiver should be able to watch all CRDs it has found.

Actual Result

Error: invalid configuration: receivers::k8sobjects: resource XY not found. Valid resources are: [ helmchartconfigs configmaps rolebindings etc. ... 

Collector version

v0.115.0

Environment information

Environment

N/A

OpenTelemetry Collector configuration

receivers:
  k8sobjects:
    objects:
      - name: telemetries
        mode: watch
        group: telemetry.istio.io/v1alpha1

Log output

Error: invalid configuration: receivers::k8sobjects: resource XY not found. Valid resources are: [helmchartconfigs configmaps rolebindings etc. ...

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/k8sobjects: @dmitryax @hvaghani221 @TylerHelmuth @ChrsMark

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[krisztianfekete]

I can see two possible approaches to handle this

• either demoting this to a warning, so the receiver can start successfully and watch available objects, or
• doing the same, but by introducing an error_mode config option, similarly (or maybe even reusing ottl.ErrorMode directly) to other components in the project, so users would be able to choose between propagating the error and ignoring missing objects

What do you think?
cc. @dmitryax, @hvaghani221, @TylerHelmuth, @ChrsMark

[TylerHelmuth]

I think a new config option is a good idea and will make sure that any changes to behavior are introduced in a non-breaking way. I think there are probably 2 modes to start:

1. Fail to start of the configured object cannot be found (existing behavior)
2. Ignore any objects that could not be found, logging a warning, and then start with whatever can be found.

I want to think about the situation in mode 2 where all objects cannot be found. Should the receiver still start?

[krisztianfekete]

I want to think about the situation in mode 2 where all objects cannot be found. Should the receiver still start?

IMO it's better to start than being in a CrashLoopBackOff. I am fine making this depend on error_mode, and only fail when it's set to propagate.

[krisztianfekete]

@TylerHelmuth, I was wondering if you could help us out here: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/38851/files#r2010626635

Today, the receiver's Validate function is checking whether any of the configured objects are missing, and if that's the case, it will fail to start.

My original approach was to introduce error_mode for validation, so users can opt-out of failing fast. @atoulme has brought up that this validation can also happen in the actual processing logic. What should be the recommended way to handle this?