Skip to content

[SPARK-8161] Set externalBlockStoreInitialized to be true, after ExternalBlockStore is initialized #6702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-8161] Set externalBlockStoreInitialized to be true, after ExternalBlockStore is initialized #6702

wants to merge 1 commit into from

Conversation

@shimingfei
Copy link

@shimingfei shimingfei commented Jun 8, 2015

externalBlockStoreInitialized is never set to be true, which causes the blocks stored in ExternalBlockStore can not be removed.

@zsxwing
Copy link
Member

zsxwing commented Jun 8, 2015

Good catch. LGTM

@SparkQA
Copy link

SparkQA commented Jun 8, 2015

Test build #34429 has finished for PR 6702 at commit add61d8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@andrewor14
Copy link
Contributor

andrewor14 commented Jun 8, 2015

@aarondav

@squito
Copy link
Contributor

squito commented Jun 9, 2015

should we add some tests in BlockManagerSuite to prevent future regressions? There is already a test "removing block", but it doesn't check that the blocks are actually removed in the underlying stores.

@shimingfei
Copy link
Author

shimingfei commented Jun 10, 2015

@squito
this bug only exists in ExternalBlockStore, which is pluggable. And to test it, we need to run the external storage system locally in the test case, which is a bit complicated.

@shimingfei
Copy link
Author

shimingfei commented Jun 15, 2015

ping....

@shimingfei
Copy link
Author

shimingfei commented Jun 16, 2015

@andrewor14 @aarondav
Can you take a look? Thanks!

@shimingfei
Copy link
Author

shimingfei commented Jun 17, 2015

ping .......

@zsxwing
Copy link
Member

zsxwing commented Jun 17, 2015

@shimingfei I think people are busy with Spark Summit.

@shimingfei
Copy link
Author

shimingfei commented Jun 17, 2015

OK, I see. Thanks.

@andrewor14
Copy link
Contributor

andrewor14 commented Jun 17, 2015

LGTM. I agree we should have a regression test, but we shouldn't block the fix on it. The code before was clearly wrong since we never set the flag to true.

@aarondav
Copy link
Contributor

aarondav commented Jun 17, 2015

LGMT

@andrewor14
Copy link
Contributor

andrewor14 commented Jun 17, 2015

Merging into master 1.4

asfgit pushed a commit that referenced this pull request Jun 17, 2015
[SPARK-8161] Set externalBlockStoreInitialized to be true, after Exte…
d75c53d 
…rnalBlockStore is initialized

externalBlockStoreInitialized is never set to be true, which causes the blocks stored in ExternalBlockStore can not be removed.

Author: Mingfei <[email protected]>

Closes #6702 from shimingfei/SetTrue and squashes the following commits:

add61d8 [Mingfei] Set externalBlockStoreInitialized to be true, after ExternalBlockStore is initialized

(cherry picked from commit 7ad8c5d)
Signed-off-by: Andrew Or <[email protected]>
@asfgit asfgit closed this in 7ad8c5d Jun 17, 2015
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-8161] Set externalBlockStoreInitialized to be true, after Exte…
47025a0 
…rnalBlockStore is initialized

externalBlockStoreInitialized is never set to be true, which causes the blocks stored in ExternalBlockStore can not be removed.

Author: Mingfei <[email protected]>

Closes apache#6702 from shimingfei/SetTrue and squashes the following commits:

add61d8 [Mingfei] Set externalBlockStoreInitialized to be true, after ExternalBlockStore is initialized
@shimingfei shimingfei deleted the SetTrue branch October 19, 2015 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@shimingfei @zsxwing @SparkQA @andrewor14 @squito @aarondav