-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-2315] Implement drop, dropRight and dropWhile for RDDs, which #1839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Can one of the admins verify this patch? |
|
This is a reboot of: |
|
Jenkins, this is okay to test. |
|
Jenkins still not getting the memo. How strict is Jenkins with commands? Is 'okay' same as 'ok'? |
take RDD as input and return new RDD with elements dropped. These methods are now implemented as lazy RDD transforms.
|
Assuming this is correct, "okay" is not same as "ok":
https://groups.google.com/forum/#!msg/quicksilver---development/Bn7RPYqAfTI/cQ-_u1BbMEQJ |
|
Jenkins, this is ok to test. Jenkins, test this please. |
|
Can one of the admins verify this patch? |
|
QA tests have started for PR 1839 at commit
|
|
QA tests have finished for PR 1839 at commit
|
|
Test PASSed. |
|
Test build #22578 has started for PR 1839 at commit
|
|
Test build #22578 has finished for PR 1839 at commit
|
|
Test PASSed. |
|
Have any admins verified this patch? |
|
@erikerlandson What do you think about releasing this (and maybe #1909) as a library on Maven or http://spark-packages.org? I'm not sure that this is an API that we necessarily want to put in core yet, but if you publish it as a package then folks would be able to use it with their existing Spark deployments without having to upgrade. The interface for users could still be pretty nice: just add an implicit class / object or set of implicit conversions, then have users import that. Spark Packages has a helpful command line tool for creating a project template, which might be a timesaver if you decide to go this route: http://spark-packages.org/package/databricks/spark-package-cmd-tool. |
|
Hi @JoshRosen, publishing some of these odds and ends in some form has been on my to-do list for a while. If there's interest, I can bump it up in priority. |
|
@JoshRosen @erikerlandson That would be great. |
|
@AlexNisnevich |
|
Hey @erikerlandson, since I don't think we're going to merge this functionality into core right now, do you mind closing this issue? BTW, it would be cool to list Silex on http://spark-packages.org, since that would put the library in front of a lot more users / eyeballs. |
|
@JoshRosen Yes, that's fine. I'll ping @willb about listing silex on spark-packages.org. |
take RDD as input and return new RDD with elements dropped.
These methods are now implemented as lazy RDD transforms.