Skip to content

[SPARK-37894][SQL] Add trash feature to FileCommitProtocol.deleteWithJob #35188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-37894][SQL] Add trash feature to FileCommitProtocol.deleteWithJob #35188

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4850e69
[SPARK-37894][SQL] Add trash feature to FileCommitProtocol.deleteWithJob
sleep1661 Jan 13, 2022
154b09c
add a comment
sleep1661 Feb 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,10 +175,19 @@ abstract class FileCommitProtocol extends Logging {

/**
* Specifies that a file should be deleted with the commit of this job. The default
* implementation deletes the file immediately.
* implementation deletes the file immediately or moves file to trash based on whether
* the trash feature is enabled.
*
* See https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
* for the relevant trash configuration
*/
def deleteWithJob(fs: FileSystem, path: Path, recursive: Boolean): Boolean = {
fs.delete(path, recursive)
if (fs.getConf.getInt("fs.trash.interval", 0) > 0 &&
Copy link
Contributor

@cloud-fan cloud-fan Feb 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where can I find the document of fs.trash.interval? It's better to add a comment to link to the hadoop conf doc page, to explain that this conf indicates enabling the trash feature.

Copy link
Contributor Author

@sleep1661 sleep1661 Feb 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had added a comment to link hadoop conf doc page. Thanks.

Trash.moveToAppropriateTrash(fs, path, fs.getConf)) {
Copy link
Contributor

@cloud-fan cloud-fan Feb 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this function call fail? if it will then it's better to try-catch it, log the error and fallback to non-trash behavior.

Copy link
Member

@HyukjinKwon HyukjinKwon Feb 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me of #29552. I think at least the concerns raised there should be addressed here.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the trash feature was merged and reverted historically from Apache Spark repository.

Copy link
Contributor

@cloud-fan cloud-fan Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trash is a feature of Hadoop file system, and we need to think about how to integrate it with Spark. This may not be the only place to consider the trash feature.

true
} else {
fs.delete(path, recursive);
}
}

/**
Expand Down