[SPARK-24093][DStream][Minor]Make some fields of KafkaStreamWriter/In…

[merlintang]

…ternalRowMicroBatchWriter visible to outside of the classes

What changes were proposed in this pull request?

This PR is created to make relevant fields of KafkaStreamWriter and InternalRowMicroBatchWriter visible to outside of the classes.

How was this patch tested?

manual tests

[merlintang]

@jerryshao can you review this minor update ?

[jerryshao]

Simply making these fields publicly accessible seems a little weird from Spark's side. Maybe we can use reflection instead.

[AmplabJenkins]

Can one of the admins verify this patch?

[merlintang]

@jerryshao Actually, we can not use reflection to get this field information.

[gaborgsomogyi]

Why is it required at all? Making things visible without proper reason is not a good idea.

[merlintang]

@gabor. These fields are important for us the understand the spark kafka streaming data like the topic name. we can use these information to track the system status.

…

On Tue, Jun 26, 2018 at 4:52 AM Gabor Somogyi ***@***.***> wrote: Why is it required at all? Making things visible without proper reason is not a good idea. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21455 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXY-RAJjzhNWzKkXIaFGViMpWFEB0hEks5uAiBlgaJpZM4USHvN> .

[gaborgsomogyi]

Sounds weird. The topic names are available when the streaming app submitted. Why this can't be reused? Second option is reflection as jerryshao suggested.

[HyukjinKwon]

BTW, fields in class constructors is not visible without val. For instance,

class A(fieldA: String) { }

generates:

$ javap -private A.class
Compiled from "A.scala"
public class A {
  public A(java.lang.String);
}

FYI, when val is added:

$ javap -private A.class
Compiled from "A.scala"
public class A {
  private final java.lang.String fieldA;
  public java.lang.String fieldA();
  public A(java.lang.String);
}

[HeartSaVioR]

Yeah, that's what also I found interestingly on Scala. Constructor parameters can be used like fields in class but they're not exposed as fields, even private accessor. For case class we can pick up values in constructor parameters but it doesn't work with normal class.