[SPARK-20293][WEB UI][History]In the page of 'jobs' or 'stages' of history server web ui,,click the 'Go' button, query paging data, the page error

[guoxiaolongzte]

What changes were proposed in this pull request?

1.In the page of 'jobs' or 'stages' of History Server web ui,
Click on the 'Go' button, query paging data, the page error, function can not be used.

2.My environment
Browser is "Chrome 49.0.2623.75 m".
Spark version is 2.1.0.
My OS is "windows 7".
Java is "java version "1.7.0_55"
spark server os is "SUSE Linux Enterprise Server 11 SP3 (x86_64) - Kernel".
Must be History Server Web ui.

3.The reasons are as follows:
'#' was escaped by the browser as% 23.
'& CompletedStage.desc = true% 23completed', the parameter value of 'desc' becomes ' true% 23',is not 'true', causing the page to report an error. The error is as follows:
HTTP ERROR 400
Problem Access / history / app-20170411132432-0004 / stages /. Reason:
For input string: "true # completed"
Powered by Jetty: //

4.UI changes require before snapshots:
url:
http://10.43.183.120:18082/history/app-20170411132432-0004/jobs/?completedJob.prevPageSize=100&completedJob.sort=Job+Id&completedJob**.desc=true%23**completed&completedJob.page=1&completedJob.pageSize=100#completed

And, I followed the images provided in the JIRA - https://issues.apache.org/jira/secure/attachment/12862828/jobs.png
https://issues.apache.org/jira/secure/attachment/12862829/stages.png
https://issues.apache.org/jira/secure/attachment/12862826/error1.png
https://issues.apache.org/jira/secure/attachment/12862827/error2.png

5.UI changes require after snapshots:
url:
http://10.43.183.120:18082/history/app-20170411132432-0004/jobs/%2Fhistory%2Fapp-20170411132432-0004%2Fjobs%2F%3F%26completedJob.sort%3DJob+Id%26completedJob.desc%3Dtrue%23completed?completedJob.prevPageSize=100&completedJob.page=1&completedJob.pageSize=100

6.The amendments are as follows:
The URL of the accessed URL is escaped to ensure that the URL is not escaped by the browser.
please see attachment of 'https://issues.apache.org/jira/browse/SPARK-20293'.

How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull request.

[srowen]

I don't quite understand, in that you say that the problem was URL-encoding the URL, but the solution here is to URL-encode it again. Is that right? maybe you can show a more concrete example of the URL as generated by the UI, and exactly what it is interpreted as, and the error page. This isn't very clear now.

[AmplabJenkins]

Can one of the admins verify this patch?

[guoxiaolongzte]

Is this, the only way to encode, will not let the browser to escape our special characters.The page will not be error.Fixed the code again.
@srowen

[ajbozarth]

@guoxiaolongzte This seems familiar, are you using the latest version of Knox with your Spark UI?

[guoxiaolongzte]

@ajbozarth I am using the latest spark version.

[ajbozarth]

After taking another look I was mixing this bug up with another when I asked that

[ajbozarth]

On a quick git blame this is my code that seems to be broken. I'll take a more detailed look when I can in the next couple days.

[guoxiaolongzte]

So how do I deal with this PR?@ajbozarth

[guoxiaolongzte]

I just downloaded the latest spark master code to compile and install , test the problem, there are still bugs, the page is wrong.

[ajbozarth]

I attempted to recreate this with the latest code on Safari, Firefox and Chrome and everything worked fine for me. What browser/environment are you using that you see this issue?

[guoxiaolongzte]

Browser is Chrome.
Spark version is 2.1.0.
Must be History Server Web ui.
My Url:
http://10.43.183.120:18082/history/app-20170411132432-0004/jobs/?completedJob.prevPageSize=100&completedJob.sort=Job+Id&completedJob.desc=true%23completed&completedJob.page=1&completedJob.pageSize=100#completed

Excuse me,Are you testing at the History Server Web Ui?What is you testing url?

Today,I try to recreate the latest Apache master code to test the bug.

[HyukjinKwon]

I can reproduce this bug.

Chrome

Safari

+My OS is ... macOS Sierra 10.12.3.

Java is ... java version "1.8.0_45"

And I manually built this against the current master c5f1cc3
via ./build/mvn -DskipTests -Phive -Phive-thriftserver package.

and then, I ran spark.range(1).show multiple times.

And, I followed the images provided in the JIRA - https://issues.apache.org/jira/secure/attachment/12862828/jobs.png and https://issues.apache.org/jira/secure/attachment/12862829/stages.png

[HyukjinKwon]

After replacing %23 to #, it seems showing the contents correctly as below:

[HyukjinKwon]

maybe you can show a more concrete example of the URL as generated by the UI, and exactly what it is interpreted as, and the error page. This isn't very clear now.

+1. @guoxiaolongzte do you mind if I ask to fill up the PR description with them? I think UI changes require before/after snapshots.

[ajbozarth]

Thanks for the followup @guoxiaolongzte and @HyukjinKwon I'll take another look at work tomorrow. Also for clarification, the reason I'm being extra detailed about this is because I want to make sure this fix isn't just a band-aid solution.

[guoxiaolongzte]

@ajbozarth @HyukjinKwon I have added the PR description again.

[ajbozarth]

So I'm still not able to recreate this, I'm unsure how the Go button is creating the url you have, the issue here is for some reason #completed is being inserted in the middle of the url where it's not supposed to be and the extra # is being encoded and causing you issues. I'm not sure this is a good fix, but we should figure out why your history server is adding the extra #completed to the middle of your url and not mine and fix that.

In order to help me figure out why our results are difference could you do an inspect on the Go button on your stage page and attach a screenshot? This is what mine looks like:

[HyukjinKwon]

@ajbozarth, I am terribly sorry. I just realised that my classpath was pointing out Spark 2.1.0.... (I was testing regressions.....).

I just clean/package again and then retested this. I can't reproduce this as below in the current master

At that time it was as below:

Spark 2.1.0

Before GO button

After GO button

Go button inspection

I double-checked %23 does not appear in the inspection appearntly.

Actual URL path after clicking the button:

jobs/?completedJob.prevPageSize=100&completedJob.sort=Job+Id&completedJob.desc=true%23completed&completedJob.page=1&completedJob.pageSize=100#completed

I clicked 검사 (meaning inspection) on the GO button below:

Current Master (7536e28)

Before GO button

After GO button

Go button inspection

I can't reproduce this now. I think we should close this. I apologise again ...

[guoxiaolongzte]

I am trying hard to test it in apache master.

[guoxiaolongzte]

@HyukjinKwon
Now build the apache master is error.

[INFO] Compiling 73 Java sources to /home/spark_build/spark-2.1.0/common/network-common/target/scala-2.11/classes...
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:144: 错误: 非法的表达式开始
[ERROR] channel.writeAndFlush(new ChunkFetchRequest(streamChunkId)).addListener(future -> {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:144: 错误: 非法的表达式开始
[ERROR] channel.writeAndFlush(new ChunkFetchRequest(streamChunkId)).addListener(future -> {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:144: 错误: 需要';'
[ERROR] channel.writeAndFlush(new ChunkFetchRequest(streamChunkId)).addListener(future -> {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:145: 错误: 不是语句
[ERROR] if (future.isSuccess()) {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:145: 错误: 需要';'
[ERROR] if (future.isSuccess()) {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:151: 错误: 有 'if', 但是没有 'else'
[ERROR] } else {
[ERROR] ^
[ERROR] /home/spark_build/spark-2.1.0/common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java:163: 错误: 非法的类型开始
[ERROR] });
[ERROR] ^

[ajbozarth]

After seeing @HyukjinKwon inspector I tracked down the issue and found #16953 that fixed this in master and 2.1 already. It was such a tiny pr I forgot all about it.

[HyukjinKwon]

@guoxiaolongzte I assume your Java version is 7. I think we dropped the support of Java 7 (https://issues.apache.org/jira/browse/SPARK-19493). We are able to build this on Windows but there are additional settings needed (e.g., memory for maven).

(FYI, I was able to build Spark on Windows 7 with 32 bits before and we are building it Windows server 2012 64 bits via AppVeyor up to my knowledge.)

@ajbozarth, thank you for your time and effort. I at least confirm that this issue is not reproducible in the current master. @guoxiaolongzte Let's close this.

[guoxiaolongzte]

@ajbozarth
@HyukjinKwon
OK.thanks. I will close this PR.

[guoxiaolongzte]

@HyukjinKwon
Thank you again, and now windows machines can compile apache master.
No my java version is 1.8.

[guoxiaolongzte]

@HyukjinKwon
I just replaced the spark-core_2.11-2.2.0-SNAPSHOT.jar
Successfully start history server, but I click the Go button, now is error, the error log is as follows:.

17/04/14 13:21:33 WARN ServletHandler: /history/app-20170411132432-0004/1/jobs/
scala.MatchError: java.lang.NoSuchMethodError: org.apache.spark.scheduler.TaskInfo.accumulables()Lscala/collection/mutable/ListBuffer; (of class java.lang.NoSuchMethodError)
at org.apache.spark.deploy.history.HistoryServer.org$apache$spark$deploy$history$HistoryServer$$loadAppUi(HistoryServer.scala:230)
at org.apache.spark.deploy.history.HistoryServer$$anon$1.doGet(HistoryServer.scala:86)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:479)
at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.spark_project.jetty.server.Server.handle(Server.java:499)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)

[HyukjinKwon]

It sounds a different issue with this. If it is a question, maybe, it might be better to ask a question to mailing list first before opening a JIRA. If it is really an issue, it might be better to report it with description, snaptops and steps to reproduce like I did above #17608 (comment) and #17608 (comment)

It seems a warning not an error BTW.