Skip to content

[SPARK-20224][SS] Updated docs for streaming dropDuplicates and mapGroupsWithState #17539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-20224][SS] Updated docs for streaming dropDuplicates and mapGroupsWithState #17539

wants to merge 3 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented Apr 5, 2017

What changes were proposed in this pull request?

  • Fixed bug in Java API not passing timeout conf to scala API
  • Updated markdown docs
  • Updated scala docs
  • Added scala and Java example

How was this patch tested?

Manually ran examples.

@SparkQA
Copy link

SparkQA commented Apr 5, 2017

Test build #75544 has finished for PR 17539 at commit e1f25b6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • public final class JavaStructuredSessionization
  • public static class LineWithTimestamp implements Serializable
  • public static class Event implements Serializable
  • public static class SessionInfo implements Serializable
  • public static class SessionUpdate implements Serializable
  • case class Event(sessionId: String, timestamp: Timestamp)
  • case class SessionInfo(
  • case class SessionUpdate(

@SparkQA
Copy link

SparkQA commented Apr 5, 2017

Test build #75545 has finished for PR 17539 at commit 33bfdcb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

brkyvz
brkyvz reviewed Apr 5, 2017
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
outputEncoder: Encoder[U],
timeoutConf: GroupStateTimeout): Dataset[U] = {
mapGroupsWithState[S, U](
mapGroupsWithState[S, U](timeoutConf)(
Copy link
Contributor

@brkyvz brkyvz Apr 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

brkyvz
brkyvz reviewed Apr 5, 2017
View reviewed changes
...ples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java Outdated
public void setExpired(boolean expired) { this.expired = expired; }
}
}

Copy link
Contributor

@brkyvz brkyvz Apr 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra lines

brkyvz
brkyvz reviewed Apr 5, 2017
View reviewed changes
docs/structured-streaming-programming-guide.md Outdated

// With watermark using guid and eventTime columns
streamingDf
.withWatermark("eventTime", "10 seconds")
Copy link
Contributor

@brkyvz brkyvz Apr 5, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing \ at end of line here and above

@brkyvz
Copy link
Contributor

brkyvz commented Apr 5, 2017

LGTM

@SparkQA
Copy link

SparkQA commented Apr 5, 2017

Test build #75556 has finished for PR 17539 at commit c78ebe8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in 9543fc0 Apr 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@brkyvz brkyvz brkyvz left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tdas @SparkQA @brkyvz