-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-14914] Fix Command too long for windows. Especially for test cases. #12694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Can one of the admins verify this patch? |
| * Create a jar file at the given path, containing a manifest with a classpath | ||
| * that references all specified entries. | ||
| */ | ||
| def createShortClassPath(tempDir: File, classPath: String) : String = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe, def createShortClassPath(tempDir: File, classPath: String): String = {
|
I'm not a fan of this approach. This means that every invocation of the launcher code will create temp files and have to deal with cleaning it up; except the launcher cannot do it, because the jvm running SparkSubmit needs it. And now SparkSubmit needs to know to clean it up. That's adding a lot of complexity. Setting the driver / executor classpaths in tests should be mostly unnecessary at this point, since the root pom file sets For the generic use case it is possible for someone could create a command line that exceeds the limit; but I think a different approach would be better. Maybe generating a temp config file, with the contents of both the user-provided config and the defaults from Spark's conf dir, but that has the same problem with "who deletes the temp conf file". On a side note, there's a ton of style issues with your patch, please try to follow the style of the code you're modifying. |
|
Update: I thought about using argument files, but they're only supported by javac (
|
|
Let's close this PR |
Closes apache#10995 Closes apache#13658 Closes apache#14505 Closes apache#14536 Closes apache#12753 Closes apache#14449 Closes apache#12694 Closes apache#12695 Closes apache#14810
What changes were proposed in this pull request?
Windows has a limitation of 8192 characters on command line. This limitation will fail the test cases related to command builder in some test cases. This pull request adopt the method in Hadoop, which create a Jar that ref all the dependencies in the manifest to shorten the class path passing to the command line.
How was this patch tested?
Unit tests on windows and Linux. Note that this commit can't fix all the unit test errors on windows.