[SPARK-45120][SPARK-45150][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI

[yaooqinn]

What changes were proposed in this pull request?

This PR is a follow-up of SPARK-45120 to cover the missing cases, and then fix test failures reported by SPARK-45150

It also brings back #42879

Why are the changes needed?

bugfix

Does this PR introduce any user-facing change?

no

How was this patch tested?

tested with the excluded CI tests bellow

[info] ChromeUISeleniumSuite:
Starting ChromeDriver 117.0.5938.62 (25a7172909a4cba7355365cf424d7d7eb35231f4-refs/branch-heads/5938@{#1146}) on port 12624
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[info] - SPARK-31534: text for tooltip should be escaped (1 second, 979 milliseconds)
[info] - SPARK-31882: Link URL for Stage DAGs should not depend on paged table. (683 milliseconds)
[info] - SPARK-31886: Color barrier execution mode RDD correctly (304 milliseconds)
[info] - Search text for paged tables should not be saved (1 second, 307 milliseconds)
[info] Run completed in 6 seconds, 702 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

no

[yaooqinn]

Hi @HyukjinKwon can you share you aqe case here or help verify this patch.?

FYI, select avg(id) from range(1E8) group by id % 5; I tested it locally and the aqe plan look ok

[HyukjinKwon]

@timarmstrong mind reviewing this please when you find some time.

[HyukjinKwon]

I can't reproduce in OSS version therefore lgtm

[yaooqinn]

Thank you @HyukjinKwon for the check :)

[yaooqinn]

thank you all, merged to master