[SPARK-49656][SS] Add support for state variables with value state collection types and read change feed options

[anishshri-db]

What changes were proposed in this pull request?

Add support for state variables with value state collection types and read change feed options

Why are the changes needed?

Without this, we cannot support reading per key changes for state variables used with stateful processors.

Does this PR introduce any user-facing change?

Yes

Users can now query value state variables with the following query:

        val changeFeedDf = spark.read
            .format("statestore")
            .option(StateSourceOptions.PATH, <checkpoint_loc>)
            .option(StateSourceOptions.STATE_VAR_NAME, <state_var_name>)
            .option(StateSourceOptions.READ_CHANGE_FEED, true)
            .option(StateSourceOptions.CHANGE_START_BATCH_ID, 0)
            .load()

How was this patch tested?

Added unit tests

[info] Run completed in 17 seconds, 318 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[anishshri-db]

cc - @jingz-db @HeartSaVioR - PTAL, thx !

[HeartSaVioR]

First pass. Mostly style and suggestion for refactor.

[HeartSaVioR]

+1

[HeartSaVioR]

Thanks! Merging to master.