[SPARK-30599][CORE][TESTS] Increase the maximum number of log events in LogAppender

[MaxGekk]

What changes were proposed in this pull request?

Increased the limit for log events that could be stored in SparkFunSuite.LogAppender from 100 to 1000.

Why are the changes needed?

Sometimes (see traces in SPARK-30599) additional info is logged via log4j, and appended to LogAppender. For example, unusual log entries are:

[36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.8 MiB)
[37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.9 MiB)
[38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2003.9 MiB)
[39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB)
[40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB)

and a test which uses LogAppender can fail with the exception:

java.lang.IllegalStateException: Number of events reached the limit of 100 while logging CSV header matches to schema w/ enforceSchema.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By re-running "SPARK-23786: warning should be printed if CSV header doesn't conform to schema" in a loop.

[srowen]

Jenkins test this please

[SparkQA]

Test build #117196 has finished for PR 27312 at commit ed4578d.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class LogAppender(msg: String = \"\", maxEvents: Int = 1000) extends AppenderSkeleton

[MaxGekk]

jenkins, retest this, please

[SparkQA]

Test build #117197 has finished for PR 27312 at commit ed4578d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class LogAppender(msg: String = \"\", maxEvents: Int = 1000) extends AppenderSkeleton

[dongjoon-hyun]

+1, LGTM. Merged to master.
Thank you, @MaxGekk and @srowen .

[dongjoon-hyun]

BTW, do we need this in branch-2.4, too?

[SparkQA]

Test build #117201 has finished for PR 27312 at commit ed4578d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class LogAppender(msg: String = \"\", maxEvents: Int = 1000) extends AppenderSkeleton

[maropu]

late LGTM;

This common subclass has been implemented recently in 88fc8db.
In branch-2.4, since the existing unit tests has the their own log appender instances, I think its less worth backporting this.

[dongjoon-hyun]

Got it. Thank you for confirmation, @maropu !