[connector/routing] When matching multiple conditions, build a new consumer each time.

[djaglowski]

Related question which can be a separate issue. Is the other (match multiple routes) approach implemented correctly? I am wondering if we are not properly fanning out the data.

For example, if we match routes 1 and 2, we cannot send the same copy of the data to both consumers. We need to use a fanout. In other words, we should build a consumer which contains all the pipelines in both 1 and 2, and send to that.

Originally posted by @djaglowski in #28888 (comment)

[github-actions]

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for connector/routing: @jpkrohling @mwear. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

@mwear, are you interested in taking a look at this one?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

I've been working on the routing connector lately and revisiting this issue I believe it is a very difficult one to solve performantly. When using match_once: false, we currently evaluate all logs against all routes and track the results of the evaluation by grouping matches together as we find them. In order to properly fan out the data, we need to completely rework the tracking mechanism so that it also remembers which set of routes were matched for each log entry.

I'm a bit skeptical that this is really worth it but I'd like to hear from others. Does anyone actually rely on match_once: false? Can we deprecate the option and have the router always act like a proper switch statement?

For what it's worth, I believe it's technically possible to achieve the match_once: false behavior by composing routers with match_once: true. Consider a case where logs are routed based on two independent dimensions:

routing:
  match_once: false
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: attributes["env"] == "dev"
       pipelines: [ logs/dev ]
    - condition: attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: attributes["region"] == "west"
       pipelines: [ logs/west ]

If there is no default case that needs to be handled, than the user can route the logs to two separate routers:

routing/env:
  match_once: true
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: attributes["env"] == "dev"
       pipelines: [ logs/dev ]
routing/region:
  match_once: true
  table:
    - condition: attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: attributes["region"] == "west"
       pipelines: [ logs/west ]

If there is a default case, then it is a bit trickier, but the user can enumerate the combination of cases:

routing:
  match_once: true
  table:
    - condition: attributes["env"] == "prod" and attributes["region"] == "east"
       pipelines: [ logs/prod, logs/east ]
    - condition: attributes["env"] == "prod" and attributes["region"] == "west"
       pipelines: [ logs/prod, logs/west ]
    - condition: attributes["env"] == "dev" and attributes["region"] == "east"
       pipelines: [ logs/dev, logs/east ]
    - condition: attributes["env"] == "dev" and attributes["region"] == "west"
       pipelines: [ logs/dev, logs/west ]
  default_pipelines: [ logs/default ]

This may not be scaleable if there are many dimensions, but another way to solve would be to place a "default router" before the handoff to the one-dimension routers. This "default router" would have almost exactly the same config as the the original match_once: false router:

routing:
  match_once: true
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/env, logs/region ] # forward to both "routing/env" and "routing/region"
    - condition: attributes["env"] == "dev"
       pipelines: [ logs/env, logs/region ]
    - condition: attributes["region"] == "east"
       pipelines: [ logs/env, logs/region ]
    - condition: attributes["region"] == "west"
       pipelines: [ logs/env, logs/region ]
  default_pipelines: [ logs/default ] # handle logs that matched no dimensions
routing/env:
  match_once: true
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: attributes["env"] == "dev"
       pipelines: [ logs/dev ]
routing/region:
  match_once: true
  table:
    - condition: attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: attributes["region"] == "west"
       pipelines: [ logs/west ]

Let's hear from others first, but if there are no objections I propose we should deprecate the option by approximating a feature gate:

• vN: Document the deprecation and add a warning log to the connector.
• vN+1: Switch the default to true.
• vN+2: Remove the setting and associated logic.

[sirianni]

Does anyone actually rely on match_once: false?

My team relies on match_once: false. We have many use cases for telemetry in our platform and different "sinks" supporting those use cases. We use the routingconnector to selectively send telemetry to those sinks. A single resource may provide telemetry that's useful in multiple sinks.

[djaglowski]

Thanks @sirianni. What do you think about the alternatives that I proposed above? If you want to share an example configuration, I would be interested in trying to convert it to produce the same result as match_once: true.