Evaluation in Python does not appear to match evaluation done via web

[jeffdn]

When utilizing a custom attribute that contains an array of strings, it is possible in the web client to configure a check against that attribute for set intersection, using is one of. For instance (these UUIDs have been randomly generated):

Any variation check for a user who has one of those UUIDs in their entity_ids custom attribute will return true, otherwise the served variation will be false. I have tried to replicate this behavior on the backend by reaching into the guts of TestData. I have tried to execute this behavior using the Files.new_data_source functionality. I cannot get it to work correctly. Am I missing something here? I'd like to be able to allow our users to test that their variations work correctly.

Specifically, a large use case for our checks will be "is the user in question associated with one or more of the entities in the list of targeted entity UUIDs?", for which we provide a body like this (all data is faked):

{
    "key": "xxxxxx",
    "custom": {
        "entity_ids": [
            "5fe43839-e535-4fcd-a12f-9202938bee18",
            "eeafcf8e-d0bd-4bec-8473-14751e2ef54b",
            "e5ae8f51-434a-4e9f-b7be-cdd64c9f5dfd"
        ]
    }
}

This works successfully when testing against the web client, but for both TestData and Files.new_data_source, even when tinkering with the guts of the LaunchDarkly code, I have been unable to get a successful execution.

SDK version: launchdarkly-server-sdk==7.5.1
language version: python-3.8.13
OS/platform: macOS 12.6

[jmhodges-color]

It looks like the problem might be that the flag-values.json generated by the web that includes a flag using that kind of targeting sets the clauses to one that uses the in op (see JSON below), but the Python definition of in op is always doing a full compare, not a scan.

I would have expected the targeting rules generated to have an "op" of is one of and for this to be a bug about is one of not being supported in the Python SDK, but it looks like maybe the Python in needs to learn when it's being applied to a list of items?

Going from the documented operators list, I suspect the Python in was written with segments in mind only, and wasn't expecting to be picked up for the is one of case?

Clauses JSON for a entity-id rule

          "clauses": [
            {
              "contextKind": "user",
              "attribute": "entity-ids",
              "op": "in",
              "values": [
                "5fe43839-e535-4fcd-a12f-9202938bee18"
              ],
              "negate": false
            }
          ],

[eli-darkly]

You've referred several times to "the web client" and "generated by the web", but I'm unclear on the distinction you're making between that and "the Python definition".

It sounds to me like you're saying that the cases where this does not work is if you're using either TestData or Files. Am I right that if you're using neither of those, then the flags are evaluated as you would expect by the Python SDK? If so, this is not about the Python SDK having a different definition of "in"; it does not, and we are very confident in saying it does not because we have cross-platform evaluation tests that cover exactly this kind of thing.

I think what I would need in order to understand this problem report is some steps to reproduce. That is, how are you using TestData and Files?

[eli-darkly]

I mean, you've provided part of the inputs for reproducing the problem, in the form of the user properties. But the other part of the inputs would be the flag definition. If I can't see how you are defining the flag when you use TestData or file data, then I can't tell whether it is correct and should produce the same output.

[jeffdn]

@eli-darkly correct, it does not work with TestData or Files. Here are some examples of how I've tried this with TestData:

Traditional approach from documentation:

td.update(
    td.flag('per-entity-id-flag')
    .variation_for_all_users(False)
    .if_match('entity_ids', *[str(eid) for eid in enabled_for_entity_ids])
    .then_return(True)
    .on(True)
)

Modified approach based on reading how the internals work:

flag_builder = td.flag('per-entity-id-flag').variation_for_all_users(False)
rule_builder = FlagRuleBuilder(flag_builder=flag_builder)
rule_builder._clauses.append(  # pylint: disable=protected-access
    {
        'attribute': 'entity_ids',
        'op': 'in',  # I've tried this with other operators as well
        'values': [str(entity_id) for entity_id in enabled_for_entity_ids],
        'negate': False,
    }
)
td.update(rule_builder.then_return(True))

And with FIles, dumped from the web UI (yes, the key is different here -- I was testing against this in the Python console rather than a unit test, but same principle and same entity_ids custom attribute):

"example-flag": {
  "key": "example-flag",
  "on": true,
  "prerequisites": [],
  "targets": [],
  "contextTargets": [],
  "rules": [
    {
      "variation": 0,
      "id": "df0855d8-6308-4d52-a9c7-6f0e44de0287",
      "clauses": [
        {
          "contextKind": "user",
          "attribute": "entity_ids",
          "op": "in",
          "values": [
            "a68318dd-bb5a-44a3-926a-bceeb197c497"
          ],
          "negate": false
        }
      ],
      "trackEvents": false
    }
  ],
  "fallthrough": {
    "variation": 1
  },
  "offVariation": 1,
  "variations": [
    true,
    false
  ],
  "clientSideAvailability": {
    "usingMobileKey": false,
    "usingEnvironmentId": true
  },
  "clientSide": true,
  "salt": "--snip--",
  "trackEvents": false,
  "trackEventsFallthrough": false,
  "debugEventsUntilDate": null,
  "version": 5
}

[eli-darkly]

Thanks. Hmm. Well... in your example of the file data, I'm still a bit confused because it doesn't seem to correspond to what the rule you showed in your screenshot. Specifically, it only has a single value in that clause. So I'm not sure how this would be a test of the "matching one list of IDs against another list of IDs" scenario.

[jeffdn]

@eli-darkly it is still a list of IDs, it just has one member in the Files example. To be very clear, we are not trying to match two lists, we are looking to see if there is an intersection between the two lists. For example:

A user with the entity ID list of [1, 2, 3] should match targeting data with is one of [2, 4, 6]
A user with the entity ID list of [1, 3, 5] should not match targeting data with is one of [2, 4, 6]

[eli-darkly]

I understand what you're trying to do, but what I'm saying is that in your file example there, you're not really testing the same behavior. What you've written there corresponds to entity_ids is one of ["a68318dd-bb5a-44a3-926a-bceeb197c497"]. That is a valid clause, and it should produce a match for a user such as {"key": "x", "custom": {"entity_ids": ["1", "2", "a68318dd-bb5a-44a3-926a-bceeb197c497", "3"]}}; if it doesn't, then that does sound like a bug, but that would be extraordinarily odd, because the Python SDK does not use different evaluation logic depending on whether the JSON data came from a file or from LaunchDarkly. If the file data corresponds exactly to the JSON data that LaunchDarkly is providing from its flag data endpoints, then there is no mechanism I can imagine that would cause the evaluation results to be different.

So, in the file data case, I can't rule out that there is a mistake in the test inputs. I still have not seen a complete set of steps to reproduce that include all the inputs for a specific test case that produces an incorrect result. For this, that would be 1. the file data and 2. the user data. The examples you've shown so far of one or the other of those don't seem to be from the same test case.

[eli-darkly]

The TestData case is a different matter and I'm still looking through the relevant code for that.

[eli-darkly]

I'm not trying to be nitpicky, but I'm sure you can understand that if there's a problem report about the file data feature not working correctly in some situation, it is much more helpful to be able to see the actual data you're talking about rather than "evaluation doesn't work when we're trying to do this general kind of thing". It doesn't have to be actual production data of course, just "we tried it with a data file whose contents are _____ and user data _____ and got result _____".

[jmhodges-color]

No, no, you're totally right. We'll provide a full repro.

[jeffdn]

@eli-darkly working one together now -- apologies for the difficulty! I should've more narrowly scoped the question as well, because the primary concern is about the TestData check. Files got included because I was working to sanity check myself, and I likely did screw that one up somewhere. I will post a repro for the TestData case shortly.

[jmhodges-color]

Okay, here's a repro test we added to testing/integrations/test_test_data_source.py to try to reproduce this clause with if_match. It's also in a branch of ours at a72215a:

def test_is_one_of():
    td = TestData.data_source()
    # Trying to use `if_match` to reproduce a targeting clause like:
    # "clauses": [
    #  {
    #    "contextKind": "user",
    #    "attribute": "entity-ids",
    #    "op": "in",
    #    "values": [
    #      "id1",
    #      "id2"
    #    ],
    #    "negate": false
    #  }
    #]
    #
    # We've also tried `if_match('entity-ids', ['id1', 'id2'])
    flag = td.flag('is-one-of-flag').if_match('entity-ids', 'id1', 'id2').then_return(True).variation_for_all_users(False).on(True)
    td.update(flag)

    config = Config('SDK_KEY', update_processor_class = td, send_events = False)
    client = LDClient(config=config)

    eval1 = client.variation_detail('is-one-of-flag', {'key': 'user1', 'custom': {'entity-ids': ['id1']}}, False)
    assert eval1.reason['kind'] == "RULE_MATCH" # Is "FALLTHROUGH" now
    assert eval1.value == True # Is False now

[jmhodges-color]

Added another test called using numbers for the variation returns (to avoid the question of "is this really the fallthrough value or is that an artifact of TestData's implementation?") and not asserting on the reason['kind'] for clarity at a0b7fb4 in our branch

[eli-darkly]

Could you try changing this line—

flag = td.flag('is-one-of-flag').if_match('entity-ids', 'id1', 'id2').then_return(True).variation_for_all_users(False).on(True)

—to this?

flag = td.flag('is-one-of-flag').variation_for_all_users(False).if_match('entity-ids', 'id1', 'id2').then_return(True).on(True)

The ordering matters because, as per the docs, variation_for_all_users has the following effect [emphasis added]:

Sets the flag to always return the specified variation for all users. The variation is specified, targeting is switched on, and any existing targets or rules are removed.

I don't expect that that is the problem, because in your earlier test data example you did have them in the right order, but I just want to rule out any confounding factors.

[jmhodges-color]

Ah, yes, that did fix it. Thank you!

I know it's documented, but that felt like surprising behavior. We'll make sure to teach our colleagues about as we migrate their flags to this system. If there was any means coming in the future to avoid being able to get into this state accidentally (perhaps with a different API), it would be welcome.

Again, thanks so much!