Fine grain control of specifying ttl #914
Conversation
|
Thank you for submitting this pull request. Unfortunately, the automated tests for this request have failed. Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/893/ |
|
Please do not commit yet - looks like broadcast blocks are sitll getting dropped, sigh ... |
|
Thank you for submitting this pull request. All automated tests for this request have passed. Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/907/ |
|
This PR does not address cases where a block is being used by a job (or its dependencies) - which is marked 'too old'. Actively used blocks should be exempted from cleanup - but unfortunately, that will need to be done in a later time. For now, this sufficiently addresses our basic concerns - it is open for review. |
|
Any updates on this ? Without this patch, for fairly large + fast jobs, master OOM's for us. |
|
Could you kindly move the pull request to the new Spark github repo under ASF mirror: https://github.com/apache/incubator-spark Thx, Henry |
There was a problem hiding this comment.
I don't like that these take both an enum type and a string name as arguments -- let's have them take just the enum.
|
Hey Mridul, sorry for the long delay on this, I was traveling after the 0.8 release. It looks good overall but I made a few comments to reduce the changes from existing code and make it more consistent. Please send it against the new repo too as Henry pointed out. |
|
Thx Henry, done. |
|
Cool! Thanks for your help Mridul |
|
Closing this since it's now here: https://github.com/apache/incubator-spark/pull/32 |
4 participants
Existing behavior is preserved (which is referenced in .
Using an enum for now - please suggest modification if inappropriate.
This is a hack to work around the problem I discussed with @matei : namely, enable cleaner to evict shuffle/etc blocks, while ensuring that there is no side effect in other parts of spark (like eviction of broadcast variables, etc).