[SPARK-14585][ML][WIP] Provide accessor methods for Pipeline stages #12420
Conversation
|
test this please |
|
Test build #56401 has finished for PR 12420 at commit
|
|
@pravingadakh Thanks! This is what I had in mind. In your tests, could you please add Java unit tests too to make sure these are Java-friendly? Also, please check the type and throw exceptions when the type does not match. I'd recommend throwing IllegalArgumentException and specifying the expected and actual types. |
|
|
||
| /** Returns stage at index i in Pipeline */ | ||
| @Since("2.0.0") | ||
| def getStage[T <: PipelineStage](i: Int): T = getStages.apply(i).asInstanceOf[T] |
There was a problem hiding this comment.
Here and elsewhere, I would just write array(i) instead of array.apply(i)
Also, can you please document the @tparam?
|
@jkbradley I'll update the PR accordingly. |
|
By the way, we'll need to get this merged quickly for it to make 2.0. (Thanks!) |
|
@jkbradley I'll update it soon. |
|
@jkbradley I have stumbled across an issue, would really appreciate your input. Have a look at this method: How do I avoid usage of |
|
Can one of the admins verify this patch? |
| def getStages: Array[PipelineStage] = $(stages).clone() | ||
|
|
||
| /** Returns stage at index i in Pipeline */ | ||
| @Since("2.0.0") |
There was a problem hiding this comment.
Can't be since 2.0.0 at this point. Also use a @return tag in the docs.
| /** Returns all stages of this type */ | ||
| @Since("2.0.0") | ||
| def getStagesOfType[T <: PipelineStage]: Array[T] = { | ||
| getStages.collect { |
There was a problem hiding this comment.
Nit: This is probably more natural as a one-liner
|
@pravingadakh Will you be able to continue with this? Thanks! |
|
I missed the ClassTag question above. Let me take a look |
|
Well, after spending a while looking around, I haven't found a good way to write this and make it Java friendly (i.e., not use ClassTag, Type, or TypeTag). Does anyone else have ideas? I'll try to return to this before long... |
|
Hi @@pravingadakh, is this still active? |
## What changes were proposed in this pull request? This PR proposes to close PRs ... - inactive to the review comments more than a month - WIP and inactive more than a month - with Jenkins build failure but inactive more than a month - suggested to be closed and no comment against that - obviously looking inappropriate (e.g., Branch 0.5) To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below: Closes apache#11129 Closes apache#12085 Closes apache#12162 Closes apache#12419 Closes apache#12420 Closes apache#12491 Closes apache#13762 Closes apache#13837 Closes apache#13851 Closes apache#13881 Closes apache#13891 Closes apache#13959 Closes apache#14091 Closes apache#14481 Closes apache#14547 Closes apache#14557 Closes apache#14686 Closes apache#15594 Closes apache#15652 Closes apache#15850 Closes apache#15914 Closes apache#15918 Closes apache#16285 Closes apache#16389 Closes apache#16652 Closes apache#16743 Closes apache#16893 Closes apache#16975 Closes apache#17001 Closes apache#17088 Closes apache#17119 Closes apache#17272 Closes apache#17971 Added: Closes apache#17778 Closes apache#17303 Closes apache#17872 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18017 from HyukjinKwon/close-inactive-prs.
6 participants
What changes were proposed in this pull request?
This PR adds accessor methods for pipeline stages in
PipelineandPipelineModelclasses.@jkbradley Hi, please validate if this is something that you had in mind. If so, I can start on test cases for these.