[FEATURE] Make a reasonable thread pool size for listing file statuses

[yaooqinn]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Setting spark.hadoop.mapreduce.input.fileinputformat.list-status.num-threads=20 if unset in engine main might be a good choice

https://github.com/apache/incubator-kyuubi/blob/d58b2bfece423df4a1bf567a85d8b4a5998cd7fb/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala#L88-L87

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[yaooqinn]

FYI apache/spark#34792