Allow Polaris CLI to set property values that contain '='

[dennishuo]

Currently the CLI parsing is too pessimistic and throws an error if it detects '=' in the parsed value of a property. However, the split 1 semantics are already standard behavior for most parsers where the right-hand-side is allowed to contain '='.

This commonly comes up if the value is itself a comma-separated kv list or if it the value is a base64-encoded string.

[eric-maynard]

I think this is reasonable, but what about =s in the key? I initially decided to just restrict = altogether because you can always use the Python client instead of the CLI for more sophisticated scenarios like this. Another alternative would be to support JSON or some other syntax that lets us escape special characters.

[dennishuo]

I think this is reasonable, but what about =s in the key? I initially decided to just restrict = altogether because you can always use the Python client instead of the CLI for more sophisticated scenarios like this. Another alternative would be to support JSON or some other syntax that lets us escape special characters.

Yeah it's worth thinking about where to draw the line between "sophisticated" scenarios and ones that work straight out of the box. As you mention, cases that might require '=' in the key could still go all the way to the Python client.

In practical terms, there does seem to be precedent for being able to specify kv lists as the value of a property and values that use base64 encodings that would include '='. In both these cases, the '=' is present in the string precisely because of the value not being from low-cardinality enumeration of possible values. It does seem that non-enumerated keys would be much rarer, since by nature such a situation would only occur if the consuming code iterates over all keys instead of performing a point-get of a key.

[eric-maynard]

Non-enumerated keys are certainly rarer, but also impossible to detect if we make this change.

…

On Mon, Feb 17, 2025 at 7:18 PM Dennis Huo ***@***.***> wrote: I think this is reasonable, but what about =s in the key? I initially decided to just restrict = altogether because you can always use the Python client instead of the CLI for more sophisticated scenarios like this. Another alternative would be to support JSON or some other syntax that lets us escape special characters. Yeah it's worth thinking about where to draw the line between "sophisticated" scenarios and ones that work straight out of the box. As you mention, cases that might require '=' in the key could still go all the way to the Python client. In practical terms, there does seem to be precedent for being able to specify kv lists as the value of a property and values that use base64 encodings that would include '='. In both these cases, the '=' is present in the string precisely because of the value not being from low-cardinality enumeration of possible values. It does seem that non-enumerated keys would be much rarer, since by nature such a situation would only occur if the consuming code iterates over all keys instead of performing a point-get of a key. — Reply to this email directly, view it on GitHub <#1003 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRE3SGNQLRWUALW7ACYZJD2QKRBTAVCNFSM6AAAAABXDWUDTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRUGUYTINBUGU> . You are receiving this because your review was requested.Message ID: ***@***.***> [image: dennishuo]*dennishuo* left a comment (apache/polaris#1003) <#1003 (comment)> I think this is reasonable, but what about =s in the key? I initially decided to just restrict = altogether because you can always use the Python client instead of the CLI for more sophisticated scenarios like this. Another alternative would be to support JSON or some other syntax that lets us escape special characters. Yeah it's worth thinking about where to draw the line between "sophisticated" scenarios and ones that work straight out of the box. As you mention, cases that might require '=' in the key could still go all the way to the Python client. In practical terms, there does seem to be precedent for being able to specify kv lists as the value of a property and values that use base64 encodings that would include '='. In both these cases, the '=' is present in the string precisely because of the value not being from low-cardinality enumeration of possible values. It does seem that non-enumerated keys would be much rarer, since by nature such a situation would only occur if the consuming code iterates over all keys instead of performing a point-get of a key. — Reply to this email directly, view it on GitHub <#1003 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRE3SGNQLRWUALW7ACYZJD2QKRBTAVCNFSM6AAAAABXDWUDTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRUGUYTINBUGU> . You are receiving this because your review was requested.Message ID: ***@***.***>

[dennishuo]

I think we should just go for consistency with common tools, and everything else by default behaves with "split at first '='":

$ ~/spark-3.5.3-bin-hadoop3/bin/spark-shell --conf spark.foo=k1=v1,k2=v2 --conf spark.bar=sdjkfh=123sdf89234=2q348usdf=/123
scala> sc.getConf.get("spark.foo")
res2: String = k1=v1,k2=v2

scala> sc.getConf.get("spark.bar")
res3: String = sdjkfh=123sdf89234=2q348usdf=/123

[eric-maynard]

This behavior is useful to users, and I'm convinced by the argument that splitting on the first = is enough of a standard. Like I said elsewhere, users who need other behavior can always use the python client.