[SPARK-49677][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset

[anishshri-db]

What changes were proposed in this pull request?

Ensure that changelog files are written on commit and forceSnapshot flag is also reset

Why are the changes needed?

Without these changes, we are not writing the changelog files per batch and we are also trying to upload full snapshot each time since the flag is not being reset correctly

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests

Before:

[info] Run completed in 3 seconds, 438 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***

After:

[info] Run completed in 4 seconds, 155 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[ericm-db]

Do we not need to change anything in setInitialCfInfo?

[anishshri-db]

Do we not need to change anything in setInitialCfInfo?

Don't think so. We need to retain the map contents/maxId across newer versions. So only resetting forceSnapshot should be sufficient I think

[anishshri-db]

@HeartSaVioR - PTAL, thx !

[HeartSaVioR]

+1

I forgot about the rationale we have to ensure both changelog and snapshot for the same version, and realized this is needed for CDC option in state data source reader.

[HeartSaVioR]

Thanks! Merging to master.