[SPARK-12507][Streaming][Document]Expose closeFileAfterWrite and allowBatching configurations for Streaming

[zsxwing]

/cc @tdas @brkyvz

[SparkQA]

Test build #48251 has finished for PR 10453 at commit 4295137.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Let's improve the title of items like this. "Update x" is never descriptive

[BenFradet]

I'd say on the driver instead of in driver.

[BenFradet]

I have a few comments on phrasing but otherwise it lgtm

[zsxwing]

for me: the default value is true.

That's why I want to expose this one since the behavior is different from 1.5.0.

[zsxwing]

@BenFradet Addressed. Thanks for your reviewing.

[SparkQA]

Test build #48436 has finished for PR 10453 at commit bce7a29.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

LGTM

[brkyvz]

Maybe we can also include that allowBatching is not just helpful when closeFileAfterWrite is enabled, but is also very helpful to scale to a large number of receivers 50+ for example.

[zsxwing]

Maybe we can also include that allowBatching is not just helpful when closeFileAfterWrite is enabled, but is also very helpful to scale to a large number of receivers 50+ for example.

I guess enabled should be disabled? Oops, ... misunderstood...

[SparkQA]

Test build #48516 has finished for PR 10453 at commit 7d9b038.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

@zsxwing Thanks! LGTM

[tdas]

"Because S3 .... " --> Set this to 'true' when you want to use S3 (or any file system that does not support flushing) for the metadata WAL at the driver.

[SparkQA]

Test build #48971 has finished for PR 10453 at commit 4d55b03.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

nit: Write Ahead Logs is not in caps in this text. so please be consistent.

[tdas]

just one more comment. then LGTM.

[SparkQA]

Test build #48980 has finished for PR 10453 at commit 28a750d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM. Merging this to master and 1.6. Thanks!