[SPARK-5353] Log failures in REPL class loading

[gzm0]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This shouldn't be debug level? Or is it a common case? What is the reference to a class server?

[gzm0]

This happens whenever a class is loaded which was not defined in the REPL (Spark Shell). But it could theoretically be an error (e.g. whole directory not available). The class server is the server that serves the classes (i.e. the actual .class files) defined in the REPL.

[srowen]

OK, it's rarely a remote server. I'd just log the URI.
Also this is borderline non-trivial enough to need a JIRA, although, I think this doesn't change any behavior, just logs things.

[gzm0]

What do you mean by just the URI? Remove the string "REPL class server at"?

I'll gladly open a JIRA, if you think it is necessary.

[srowen]

Yes exactly. The URI is typically not remote, not from some server, just a local JAR.

[gzm0]

It is actually a remote URI: Have a look at the usage site. It can either point to an HTTP server or an HDFS.

[gzm0]

Just added a JIRA

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #25927 has started for PR 4130 at commit 4fa0582.

• This patch merges cleanly.

[SparkQA]

Test build #25927 has finished for PR 4130 at commit 4fa0582.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • logDebug(s"Did not load class $name from REPL class server at $uri", e)
  • logError(s"Failed to check existence of class $name on REPL class server at $uri", e)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25927/
Test FAILed.

[gzm0]

@marmbrus could you have a look at this as well, please?

[SparkQA]

Test build #567 has started for PR 4130 at commit 4fa0582.

• This patch merges cleanly.

[SparkQA]

Test build #567 has finished for PR 4130 at commit 4fa0582.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

test this please

[SparkQA]

Test build #26440 has started for PR 4130 at commit 4fa0582.

• This patch merges cleanly.

[SparkQA]

Test build #26440 has finished for PR 4130 at commit 4fa0582.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • logDebug(s"Did not load class $name from REPL class server at $uri", e)
  • logError(s"Failed to check existence of class $name on REPL class server at $uri", e)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26440/
Test PASSed.

[marmbrus]

@pwendell does this seem reasonable?

[rxin]

lgtm.

[pwendell]

LGTM

[pwendell]

I can merge it.