fix a defective cross-version module dependency, enable testing in java 17

[tribbloid]

No description provided.

[pomadchin]

Thank you!

[tribbloid]

My pleasure, tho it is slow 😭

[pomadchin]

@tribbloid do you want me to cut a release btw? 👀

[tribbloid]

@pomadchin that's not for me to decide, PMC should have their own release deadline

I'm eager to contribute to upgrading to Spark 4.0.0 & optional Scala 3.x support in the next major release. Already had a successful prototype but Spark 3.x backward compatibility is still problematic. Should I post proposal here or elsewhere (e.g. typelevel Discord channel)?

[pomadchin]

@tribbloid the best would be a ticket creation (issue); there are also lots of PRs by @chris-twiner that we might want to consider; with this one though we need some time to review the approach, and I'm not 100% convinced that moving shims into a separate library is the best approach.

However, would appreciate extra thoughts and these PR reviews; I am planning to get back to them this / next month.

In the last year did not have much to invest into frameless unfortunately :/ But Spark 4 is of the top next priority for now tbh. Should have more time in in the coming months prior to January.

[chris-twiner]

@tribbloid the best would be a ticket creation (issue); there are also lots of PRs by @chris-twiner that we might want to consider; with this one though we need some time to review the approach, and I'm not 100% convinced that moving shims into a separate library is the best approach.

However, would appreciate extra thoughts and these PR reviews; I am planning to get back to them this / next month.

In the last year did not have much to invest into frameless unfortunately :/ But Spark 4 is of the top next priority for now tbh. Should have more time in in the coming months prior to January.

@pomadchin fyi - an update on the current state of play - there are two versions of spark 4 support, one using ExpressionEncoders, the deprecated support was added into Spark 4 to allow for 4.0 migrations only and will be removed in 4.1. This path encoding approach is released in my fork, which is source and binary compat with main non Spark4 frameless.

The other version, which is source and binary incompatible, using AgnosticEncoders mostly works but has limitations (number one being positional GetOrdinal usage is no longer possible, the As 'solution' I have added passes the tests but relies on a renaming projection of top fields to work). Connect wise any local compilation such as Literal, udf, Left/Right disambiguation and Join are probably not going to work, and I am currently dev'ing Connect client usage of the derived agnostic encoding (Injections/Codecs should work as they are local to Arrow[De/S]erializer) using the testing project but that's weeks off of being proven.

With AgnosticEncoder support being source incompatible and the shift of execution model in Spark 4 pretty it may be worth making a fresh frameless that just targets Spark 4 with spark-sql-api compliance (i.e. should run on connect) and find compliant approaches to Literal, udf, left/right and join.