[Spark-2201] Improve FlumeInputDStream's stability and make it scalable

[joyyoj]

re-submit

[AmplabJenkins]

Can one of the admins verify this patch?

[harishreedharan]

AbstractRpcClient already implements RpcClient - so you don't really need to implement RpcClient.

[SparkQA]

Can one of the admins verify this patch?

[tdas]

Hey @joyyoj

This is a very cool functionality! In fact, we had added a subset of this functionality (sink) as part Spark 1.1, thus validating your approach. However, while this approach provides more functionality, it does at the cost of maintaining this, debugging this, and the cost of addition dependencies like to the spark project, like zookeeper, etc.

We are seeing a lot of such big contributions from the community, which is very encouraging and useful, but will be very hard to maintain for us. So we are trying to figure out a way by which the community can contribute such custom functionality, and maintain it themselves. Say something like a contribs repo...

I will let you know when we have figured something out. Till then please bear with us. :)

[tdas]

@joyyoj
We have set up http://spark-packages.org a way for the community to contribute features and maintain it themselves. Please consider adding your this Flume functionality to http://spark-packages.org/ and let the community use it.

Regarding this PR, mind closing it?

[tdas]

@joyyoj Mind closing this?

[joyyoj]

Never mind, very very sorry for replay so late.