Skip to content

[SPARK-8691][WebUI]Enable GZip by default for Web UI #7072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-8691][WebUI]Enable GZip by default for Web UI #7072

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,11 @@
<artifactId>jetty-servlet</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlets</artifactId>
<scope>compile</scope>
</dependency>
<!-- Because we mark jetty as provided and shade it, its dependency
orbit is ignored, so we explicitly list it here (see SPARK-5557).-->
<dependency>
Expand Down
19 changes: 19 additions & 0 deletions core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import scala.xml.Node
import org.eclipse.jetty.server.Server
import org.eclipse.jetty.server.handler._
import org.eclipse.jetty.servlet._
import org.eclipse.jetty.servlets.GzipFilter
import org.eclipse.jetty.util.thread.QueuedThreadPool
import org.json4s.JValue
import org.json4s.jackson.JsonMethods.{pretty, render}
Expand Down Expand Up @@ -166,8 +167,26 @@ private[spark] object JettyUtils extends Logging {
contextHandler
}

private def addDefaultFilters(handlers: Seq[ServletContextHandler], conf: SparkConf): Unit = {
Copy link
Member

@srowen srowen Jun 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely Jetty/netty already handle this internally? this is a hacky approach to supporting compression and I am all but certain modern containers (can be configured to) do this for you. Tomcat does.

Copy link
Member Author

@zsxwing zsxwing Jun 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is we don't have a configuration file for Jetty. So to enable it, we need to set it in the code. Here is an example provided by Jetty: https://github.com/jetty-project/jetty-plugin-support/blob/378f5f691fc24c3f223e7239fc56b3568b6f816e/jetty-servlets/src/test/java/org/eclipse/jetty/servlets/GzipWithPipeliningTest.java#L58

val compression = conf.getBoolean("spark.ui.compression", true)
Copy link
Contributor

@pwendell pwendell Jun 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to make this configurable? When would a user not want this?

Copy link
Member Author

@zsxwing zsxwing Jun 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Spark supports the user to add custom Filters, I'm concerned that they may add a Filter that conflicts with GZipFilter. So I add this configuration to disable GZipFilter.

if (compression) {
// Enable gzip for Web UI
logInfo("Adding filter: GzipFilter")
val holder = new FilterHolder()
holder.setClassName(classOf[GzipFilter].getName())
holder.setInitParameter("methods", "GET,POST")
holder.setInitParameter("mimeTypes", "text/html,text/xml,text/plain,text/css," +
"text/javascript,text/json,application/x-javascript,application/javascript," +
"application/json,application/xml,application/xml+xhtml,image/svg+xml")
val enumDispatcher = java.util.EnumSet.of(DispatcherType.ASYNC, DispatcherType.ERROR,
DispatcherType.FORWARD, DispatcherType.INCLUDE, DispatcherType.REQUEST)
handlers.foreach { case (handler) => handler.addFilter(holder, "/*", enumDispatcher) }
}
}

/** Add filters, if any, to the given list of ServletContextHandlers */
def addFilters(handlers: Seq[ServletContextHandler], conf: SparkConf) {
addDefaultFilters(handlers, conf)
val filters: Array[String] = conf.get("spark.ui.filters", "").split(',').map(_.trim())
filters.foreach {
case filter : String =>
Expand Down
7 changes: 7 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,12 @@
<version>${jetty.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlets</artifactId>
<version>${jetty.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
Expand Down Expand Up @@ -1442,6 +1448,7 @@
<include>org.eclipse.jetty:jetty-http</include>
<include>org.eclipse.jetty:jetty-continuation</include>
<include>org.eclipse.jetty:jetty-servlet</include>
<include>org.eclipse.jetty:jetty-servlets</include>
<include>org.eclipse.jetty:jetty-plus</include>
<include>org.eclipse.jetty:jetty-security</include>
<include>org.eclipse.jetty:jetty-util</include>
Expand Down