[exporter/azuremonitor] collector can't send data to different azure application insights

[emperorente]

Component(s)

exporter/azuremonitor

Describe the issue you're reporting

Hi team,

We're trying to send the data to different Application insights, use the demo under opentelemetry-collector-contrib/examples/demo/,

and also add semconv.ServiceNamespaceKey.String("GROUPB")|semconv.ServiceNamespaceKey.String("GROUPA") for client and server

Here is the config of collector:

  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch: {}
  filter/groupa:
    # error_mode: ignore
    metrics:
      metric:
        - not (resource.attributes["service.namespace"] == "GROUPA")
      # include:
        # match_type: regexp
        # metric_names:
        #   - demo_client/.*
        # match_type: expr
        # expressions:
        #   - MetricName == Label("service.namespace") == "GROUPB"
    traces:
      span:
        - not (resource.attributes["service.namespace"] == "GROUPA")
  filter/groupb:
    # error_mode: ignore
    metrics:
      metric:
        - not (resource.attributes["service.namespace"] == "GROUPB")
      # include:
      #   match_type: regexp
      #   metric_names:
      #     - demo_server/.*
    traces:
      span:
        - not (resource.attributes["service.namespace"] == "GROUPB")
      


exporters:
  azuremonitor/groupa:
    connection_string: "InstrumentationKey=xxxxxx;IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/"
    spaneventsenabled: true
    # maxbatchsize: 4096
    # maxbatchinterval: 30s
  azuremonitor/groupb:
    connection_string: "InstrumentationKey=xxxxxx;IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/"
    spaneventsenabled: true
    # maxbatchsize: 4096
    # maxbatchinterval: 30s

service:
  pipelines:
    metrics/groupb:
      receivers: [otlp]
      processors: [filter/groupb]
      exporters: [azuremonitor/groupb]
    traces/groupb:
      receivers: [otlp]
      processors: [filter/groupb]
      exporters: [azuremonitor/groupb]
    metrics/groupa:
      receivers: [otlp]
      processors: [filter/groupa]
      exporters: [azuremonitor/groupa]
    traces/groupa:
      receivers: [otlp]
      processors: [filter/groupa]
      exporters: [azuremonitor/groupa]
  telemetry:
    logs:
      level: debug

docker-compose.yml:

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "55679:55679" # zpages extension

  demo-client:
    build:
      dockerfile: Dockerfile
      context: ./client
    restart: always
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - DEMO_SERVER_ENDPOINT=http://demo-server:7080/hello
    depends_on:
      - demo-server

  demo-server:
    build:
      dockerfile: Dockerfile
      context: ./server
    restart: always
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    ports:
      - "7080"
    depends_on:
      - otel-collector

when I start it, Collector shows error as :

"2024-07-22T06:07:36.637Z debug azuremonitorexporter@v0.96.0/factory.go:139 #78 - 400 Invalid instrumentation key {"kind": "exporter", "data_type": "traces", "name": "azuremonitor/groupb"}
2024-07-22T06:07:36.637Z debug azuremonitorexporter@v0.96.0/factory.go:139 Telemetry item:
{"ver":1,"name":"Microsoft.ApplicationInsights.Metric","time":"2024-07-22T06:07:34.864590456Z","sampleRate":100,"seq":"","iKey":"","tags":{"ai.cloud.role":"demo-client","ai.internal.sdkVersion":"otelc-v0.96.0-lin-arm64"},"data":{"baseType":"MetricData","baseData":{"ver":2,"metrics":[{"name":"demo_client/line_counts","kind":0,"value":18527,"count":1,"min":0,"max":0,"stdDev":0}],"properties":{"client":"cli","host.name":"4a824b92f1b5","instrumentationlibrary.name":"demo-client-meter","method":"repl","process.command_args":"","process.executable.name":"main","process.executable.path":"/app/main","process.owner":"root","process.pid":"","process.runtime.description":"go version go1.22.5 linux/arm64","process.runtime.name":"go","process.runtime.version":"go1.22.5","service.name":"demo-client","service.namespace":"GROUPB","telemetry.sdk.language":"go","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.28.0"}}}}"

Could you please take a look at it?

[github-actions]

Pinging code owners:

• exporter/azuremonitor: @pcwiese

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[puputtiap-sympa]

We saw the same issue, azuremonitor exporter is not able to send multiple application insights if you use connection_string but somehow it does work if you use the legacy instrumentation_key instead