[SPARK-19824][Core] Update JsonProtocol to keep consistent with the UI #18303

jiangxb1987 · 2017-06-14T15:03:30Z

What changes were proposed in this pull request?

Fix any inconsistent part in JsonProtocol with the UI.
This PR also contains the modifications in #17181

How was this patch tested?

Updated JsonProtocolSuite.

Before this change, localhost:8080/json shows:

{
  "url" : "spark://xingbos-MBP.local:7077",
  "workers" : [ {
    "id" : "worker-20170615172946-192.168.0.101-49450",
    "host" : "192.168.0.101",
    "port" : 49450,
    "webuiaddress" : "http://192.168.0.101:8081",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497519481722
  }, {
    "id" : "worker-20170615172948-192.168.0.101-49452",
    "host" : "192.168.0.101",
    "port" : 49452,
    "webuiaddress" : "http://192.168.0.101:8082",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497519484160
  }, {
    "id" : "worker-20170615172951-192.168.0.101-49469",
    "host" : "192.168.0.101",
    "port" : 49469,
    "webuiaddress" : "http://192.168.0.101:8083",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497519486905
  } ],
  "cores" : 24,
  "coresused" : 24,
  "memory" : 46080,
  "memoryused" : 3072,
  "activeapps" : [ {
    "starttime" : 1497519426990,
    "id" : "app-20170615173706-0001",
    "name" : "Spark shell",
    "user" : "xingbojiang",
    "memoryperslave" : 1024,
    "submitdate" : "Thu Jun 15 17:37:06 CST 2017",
    "state" : "RUNNING",
    "duration" : 65362
  } ],
  "completedapps" : [ {
    "starttime" : 1497519250893,
    "id" : "app-20170615173410-0000",
    "name" : "Spark shell",
    "user" : "xingbojiang",
    "memoryperslave" : 1024,
    "submitdate" : "Thu Jun 15 17:34:10 CST 2017",
    "state" : "FINISHED",
    "duration" : 116895
  } ],
  "activedrivers" : [ ],
  "status" : "ALIVE"
}

After the change:

{
  "url" : "spark://xingbos-MBP.local:7077",
  "workers" : [ {
    "id" : "worker-20170615175032-192.168.0.101-49951",
    "host" : "192.168.0.101",
    "port" : 49951,
    "webuiaddress" : "http://192.168.0.101:8081",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497520292900
  }, {
    "id" : "worker-20170615175034-192.168.0.101-49953",
    "host" : "192.168.0.101",
    "port" : 49953,
    "webuiaddress" : "http://192.168.0.101:8082",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497520280301
  }, {
    "id" : "worker-20170615175037-192.168.0.101-49955",
    "host" : "192.168.0.101",
    "port" : 49955,
    "webuiaddress" : "http://192.168.0.101:8083",
    "cores" : 8,
    "coresused" : 8,
    "coresfree" : 0,
    "memory" : 15360,
    "memoryused" : 1024,
    "memoryfree" : 14336,
    "state" : "ALIVE",
    "lastheartbeat" : 1497520282884
  } ],
  "aliveworkers" : 3,
  "cores" : 24,
  "coresused" : 24,
  "memory" : 46080,
  "memoryused" : 3072,
  "activeapps" : [ {
    "id" : "app-20170615175122-0001",
    "starttime" : 1497520282115,
    "name" : "Spark shell",
    "cores" : 24,
    "user" : "xingbojiang",
    "memoryperslave" : 1024,
    "submitdate" : "Thu Jun 15 17:51:22 CST 2017",
    "state" : "RUNNING",
    "duration" : 10805
  } ],
  "completedapps" : [ {
    "id" : "app-20170615175058-0000",
    "starttime" : 1497520258766,
    "name" : "Spark shell",
    "cores" : 24,
    "user" : "xingbojiang",
    "memoryperslave" : 1024,
    "submitdate" : "Thu Jun 15 17:50:58 CST 2017",
    "state" : "FINISHED",
    "duration" : 9876
  } ],
  "activedrivers" : [ ],
  "completeddrivers" : [ ],
  "status" : "ALIVE"
}

jiangxb1987 · 2017-06-14T15:03:47Z

cc @yongtang @gatorsmile @cloud-fan

SparkQA · 2017-06-14T17:08:38Z

Test build #78054 has finished for PR 18303 at commit 244bbae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-14T19:17:22Z

Show the screenshot before and after this PR?

jiangxb1987 · 2017-06-15T00:00:53Z

@gatorsmile Sure, let me do it later!

jiangxb1987 · 2017-06-15T09:55:01Z

retest please.

cloud-fan · 2017-06-16T04:46:18Z

retest this please

cloud-fan · 2017-06-16T04:50:08Z

LGTM

SparkQA · 2017-06-16T06:46:53Z

Test build #78154 has finished for PR 18303 at commit 244bbae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-06-16T06:58:07Z

retest this please

gatorsmile · 2017-06-16T07:11:58Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

    ("duration" -> obj.duration)
  }

  def writeApplicationDescription(obj: ApplicationDescription): JObject = {


IMO, we need to add the function descriptions for these public functions and provide a clear definition for each field. How about creating a few JIRAs and asking the open source community to do it?

Let me do this at the weekend in this PR.

SparkQA · 2017-06-16T09:51:05Z

Test build #78163 has finished for PR 18303 at commit 244bbae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-18T17:16:08Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

-   ("lastheartbeat" -> obj.lastHeartbeat)
- }
+  /**
+   * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a


, a -> . A

gatorsmile · 2017-06-18T17:20:30Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *         `id` a string identifier of the worker
+   *         `host` the host that the worker is running on
+   *         `port` the port that the worker is bound to
+   *         `address` ${host}:${port}


If we already has host and port, why we still need the address?

Good point, I'll remove this.

gatorsmile · 2017-06-18T17:23:49Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *
+   * @return a Json object containing the following fields:
+   *         `id` a string identifier of the application
+   *         `starttime` time in milliseconds that the application starts


Switch the order between id and starttime?

gatorsmile · 2017-06-18T17:24:34Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   * @return a Json object containing the following fields:
+   *         `id` a string identifier of the application
+   *         `starttime` time in milliseconds that the application starts
+   *         `name` a name describes the application


-> the description of the application

gatorsmile · 2017-06-18T17:26:11Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+  }

+  /**
+   * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the


, an -> . An

gatorsmile · 2017-06-18T17:26:30Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

  }

+  /**
+   * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists


, an -> . An

gatorsmile · 2017-06-18T17:27:02Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   * of the description of an application.
+   *
+   * @return a Json object containing the following fields:
+   *         `name` a name describes the application


-> the description of the application

gatorsmile · 2017-06-18T17:27:46Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *
+   * @return a Json object containing the following fields:
+   *         `name` a name describes the application
+   *         `cores` max cores can be allocated to the application, 0 means unlimited


max cores can be -> max cores that can be

gatorsmile · 2017-06-18T17:28:31Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *         `cores` max cores can be allocated to the application, 0 means unlimited
+   *         `memoryperslave` minimal memory in MB required to each executor
+   *         `user` name of the user who submitted the application
+   *         `command` the command string that submitted the application


the command string that submitted -> the command used to submit

gatorsmile · 2017-06-18T17:29:14Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

  }

+  /**
+   * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the


, an -> . An

gatorsmile · 2017-06-18T17:29:35Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   * information of an executor.
+   *
+   * @return a Json object containing the following fields:
+   *         `id` a integer identifier of the executor


a integer -> an integer

gatorsmile · 2017-06-18T17:30:27Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   * @return a Json object containing the following fields:
+   *         `id` a integer identifier of the executor
+   *         `memory` memory in MB allocated to the executor
+   *         `appid` a string identifier of the application that the executor is working for


working for -> working on

gatorsmile · 2017-06-18T17:30:44Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *         `memory` memory in MB allocated to the executor
+   *         `appid` a string identifier of the application that the executor is working for
+   *         `appdesc` a Json object of the [[ApplicationDescription]] of the application that the
+   *         executor is working for


working for -> working on

gatorsmile · 2017-06-18T17:31:17Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

  }

+  /**
+   * Export the [[DriverInfo]] to a Json object, a [[DriverInfo]] consists of the information of a


, a -> . A

gatorsmile · 2017-06-18T17:35:11Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

  }

+  /**
+   * Export the [[MasterStateResponse]] to a Json object, a [[MasterStateResponse]] consists the


, a -> . A

gatorsmile · 2017-06-18T17:36:21Z

core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala

+   *         `activeapps` a list of Json objects of [[ApplicationInfo]] of the active applications
+   *         running on the master
+   *         `completedapps` a list of Json objects of [[ApplicationInfo]] of the completed
+   *         applications from the master


-> the applications completed in the master

gatorsmile · 2017-06-18T17:38:20Z

LGTM except a few comments.

SparkQA · 2017-06-18T18:35:16Z

Test build #78227 has finished for PR 18303 at commit 8dc5218.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-19T03:41:51Z

LGTM

cloud-fan · 2017-06-19T04:00:42Z

LGTM

SparkQA · 2017-06-19T05:00:40Z

Test build #78235 has finished for PR 18303 at commit 8c39912.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-19T05:05:22Z

Thanks! Merging to master.

update JsonProtocol

244bbae

gatorsmile reviewed Jun 16, 2017

View reviewed changes

add comments

8dc5218

gatorsmile reviewed Jun 18, 2017

View reviewed changes

update comments.

8c39912

asfgit closed this in ea542d2 Jun 19, 2017

jiangxb1987 deleted the json-protocol branch June 27, 2017 08:49

[SPARK-19824][Core] Update JsonProtocol to keep consistent with the UI #18303

[SPARK-19824][Core] Update JsonProtocol to keep consistent with the UI #18303

Uh oh!

Conversation

jiangxb1987 commented Jun 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

jiangxb1987 commented Jun 14, 2017

Uh oh!

SparkQA commented Jun 14, 2017

Uh oh!

gatorsmile commented Jun 14, 2017

Uh oh!

jiangxb1987 commented Jun 15, 2017

Uh oh!

jiangxb1987 commented Jun 15, 2017

Uh oh!

cloud-fan commented Jun 16, 2017

Uh oh!

cloud-fan commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 16, 2017

Uh oh!

cloud-fan commented Jun 16, 2017

Uh oh!

gatorsmile Jun 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jun 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jun 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jun 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jun 18, 2017

Uh oh!

SparkQA commented Jun 18, 2017

Uh oh!

gatorsmile commented Jun 19, 2017

jiangxb1987 commented Jun 14, 2017 •

edited

Loading

gatorsmile Jun 16, 2017 •

edited

Loading

gatorsmile Jun 18, 2017 •

edited

Loading

gatorsmile Jun 18, 2017 •

edited

Loading

gatorsmile Jun 18, 2017 •

edited

Loading