[SPARK-32481][TESTS][FOLLOWUP] Use different directory name for MacOS #29550
Conversation
|
@LuciferYang I think this should be ok |
| val fs = tablePath.getFileSystem(hadoopConf) | ||
| val trashRoot = fs.getTrashRoot(tablePath) | ||
| assert(!fs.exists(trashRoot)) | ||
| val trashCurrent = new Path(fs.getTrashRoot(tablePath), "Current") |
There was a problem hiding this comment.
Can we make a fake file system (that extends FileSystem) and use a temporary directory?
There was a problem hiding this comment.
Hii @HyukjinKwon, but Trash.moveToAppropriateTrash will always move the data to trashCurrent. can you guide me on what you mean by using a temporary directory instead.
dongjoon-hyun
changed the title
|
ok to test |
|
The original commit is reverted due to the compilation error with Hadoop 2.7 profile. Since this PR get conflicted already, let's include this when we have a next main PR. |
|
Test build #127931 has finished for PR 29550 at commit
|
|
@Udbhav30 hi, Udbhav30,Could you tell me which spark version you make that modification.Because we also want recovery product data when inserting overwrite wrong data or partition , i want to do the migration that we migrate you modification to Spark 2.4.2 |
This feature was merged in spark 3.1.0 but later it was reverted back. So spark doesn't have this feature right now |
5 participants
What changes were proposed in this pull request?
This is a followup of #29387 to correct the test-case.
Why are the changes needed?
trashRootis always present in macOS, so it's better to use exacttrashPathto check if data is moved to trash.Does this PR introduce any user-facing change?
No
How was this patch tested?
Test modified