[cmd/opampsupervisor] Fix merging lists when using local config

[evan-bradley]

Description

With #38671, we supply the local config to the Collector in two places:

1. The effective (merged) config
2. A list of files passed through --config flags to the Collector.

While the second approach is preferable in the long term, we have to manually merge the config for now since the Collector doesn't support merging lists in config without a feature gate enabled. This PR therefore opts for the first option until we enable this feature gate.

Fixes #39931

[douglascamata]

I think this PR is useful. But it's solving a different problem than what's stated, imho.

[douglascamata]

@evan-bradley we might also want to change the priority order for merging configuration:

opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor/supervisor.go

Lines 1243 to 1262 in a0044fb

	// Merge own metrics config.
	ownMetricsCfg, ok := s.agentConfigOwnMetricsSection.Load().(string)
	if ok {
	if err = k.Load(rawbytes.Provider([]byte(ownMetricsCfg)), yaml.Parser(), koanf.WithMergeFunc(configMergeFunc)); err != nil {
	return false, err
	}
	}
	// Merge local config last since it has the highest precedence.
	if err = k.Load(rawbytes.Provider(s.composeExtraLocalConfig()), yaml.Parser(), koanf.WithMergeFunc(configMergeFunc)); err != nil {
	return false, err
	}
	if err = k.Load(rawbytes.Provider(s.composeOpAMPExtensionConfig()), yaml.Parser(), koanf.WithMergeFunc(configMergeFunc)); err != nil {
	return false, err
	}
	if err = k.Load(rawbytes.Provider(s.composeAgentConfigFiles()), yaml.Parser(), koanf.WithMergeFunc(configMergeFunc)); err != nil {
	return false, err
	}

This doesn't seem right to me. Local configuration (from composeAgentConfigFiles) is being merged last (on top of everything), so it can overwrite the built-in/internal configuration (from composeExtraLocalConfig, which I think we should rename to avoid confusion), the opamp configuration (from composeOpAMPExtensionConfig), and also own metrics configuration (from ownMetricsCfg).

I think local configuration should be the very first to be merged, as they shouldn't override any other configuration.

[evan-bradley]

@douglascamata Thanks for taking such a close look. I agree that we should probably make local configuration the lowest priority when merging, but the reason this was done like this is described here: #38671 (comment). I think that comment also makes some good points, and in my opinion points to the idea that this should be configurable.

For now I'd like to leave the ordering as-is and fix it in its own PR where we can consider it independently of this issue.

[douglascamata]

@evan-bradley sounds good to me. My use case for wanting to have local configuration being the lowest priority is because I want to use it as a "base layer" configuration to allow the Supervisor to start Collectors even without being able to talk to the upstream OpAMP backend.

I don't want to have a default remote configuration that is required to be delivered to all Supervisor before Collectors can start.

[douglascamata]

For the record, I created an issue for the use cas I mentioned of local configuration with the lowest priority (aka base layer config): #39963