[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query #23609

HeartSaVioR · 2019-01-22T08:20:44Z

What changes were proposed in this pull request?

This patch proposes to fix issue on adding current_timestamp / current_date with streaming query.

The root reason is that Spark transforms CurrentTimestamp/CurrentDate to CurrentBatchTimestamp in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution.
(In ContinuousExecution, Spark doesn't allow using current_timestamp and current_date so it has been OK.)

It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors.

This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely.

How was this patch tested?

Added UT.

HeartSaVioR · 2019-01-22T08:21:14Z

cc. @tdas @zsxwing @jose-torres @arunmahadevan @gaborgsomogyi

HeartSaVioR · 2019-01-22T08:23:16Z

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala

We can even leverage newBatchesPlan and remove this: this effort is to ensure going through the same path with further transformation when extracting schema.

I'm fine with this.

@HeartSaVioR . I prefer to leverage the existing lines (501 ~ 509) as you said like the following. I assume that the following is what you mean. Eventually, it's two line changes (excluding comments). WDYT?

// Rewire the plan to use the new attributes that were returned by the source. val newAttributePlan = newBatchesPlan transformAllExpressions { case ct: CurrentTimestamp => CurrentBatchTimestamp(offsetSeqMetadata.batchTimestampMs, - ct.dataType) + ct.dataType).toLiteral case cd: CurrentDate => CurrentBatchTimestamp(offsetSeqMetadata.batchTimestampMs, - cd.dataType, cd.timeZoneId) + cd.dataType, cd.timeZoneId).toLiteral }

@dongjoon-hyun
Actually I meant we pass newBatchesPlan instead of newAttributePlan only for extracting schema since we actually guess the name and data type will not be changed from IncrementalExecution. But IMHO I feel safer pre-transforming it.

Transforming CurrentBatchTimestamp to Literal not in IncrementalExecution breaks the intention what CurrentBatchTimestamp javadoc describes. Please note that I used transformed plan only for extracting schema and the actual plan is not changed.

…eaming query

SparkQA · 2019-01-22T08:26:24Z

Test build #101515 has finished for PR 23609 at commit dc09c99.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-22T08:32:19Z

Test build #101516 has finished for PR 23609 at commit afb0dc4.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-01-22T08:34:47Z

Seems like it's relevant issue:

MicroBatchExecution.scala:517: unreachable code

HeartSaVioR · 2019-01-22T08:35:37Z

@gaborgsomogyi
Yeah my bad. I guess I've fixed in 8f30a75. Wondering why my maven didn't stop the build.

gaborgsomogyi

Tested with spark-shell as well and works fine.

gaborgsomogyi · 2019-01-22T09:15:38Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

    }
  }
+
+  Seq(true, false).foreach { useV2Sink =>


This structure makes it impossible to execute test independently (at least I don't know). The whole suite has to be executed which makes even small modifications time consuming.

RIght. That was to minimize the code, but I guess only some lines will be added and it will become easier to test so better to apply your suggestion. Thanks!

gaborgsomogyi · 2019-01-22T09:16:54Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

  }
+
+  Seq(true, false).foreach { useV2Sink =>
+    import org.apache.spark.sql.functions._


Already imported.

gaborgsomogyi · 2019-01-22T09:29:13Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

+        row.getTimestamp(1).getTime
+      }
+
+      var lastTimestamp = -1L


Not much functional difference but maybe this could be now.

gaborgsomogyi · 2019-01-22T09:53:00Z

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala

I'm fine with this.

gaborgsomogyi · 2019-01-22T10:00:16Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

+
+    test(newTestName) {
+      val input = MemoryStream[Int]
+      val df = input.toDS().withColumn("cur_timestamp", lit(current_timestamp()))


As I see this covers CurrentTimestamp, maybe worth to add test for CurrentDate as well.

gaborgsomogyi

LGTM.

gaborgsomogyi · 2019-01-22T10:54:23Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

+      CheckLastBatch { rows: Seq[Row] =>
+        lastTimestamp = assertBatchOutputAndUpdateLastTimestamp(rows, lastTimestamp, currentDate, 1)
+      },
+      Execute { _ => Thread.sleep(3 * 1000) },


No strong opinion but spending 6 seconds for 2 tests maybe overkill.

Yeah good suggestion. 1s would be enough.

SparkQA · 2019-01-22T12:16:06Z

Test build #101525 has finished for PR 23609 at commit d09845b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-01-22T12:33:58Z

retest this please

SparkQA · 2019-01-22T12:35:48Z

Test build #101517 has finished for PR 23609 at commit 8f30a75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-22T12:44:27Z

Test build #101527 has finished for PR 23609 at commit 687c3e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-01-22T12:49:06Z

retest this please

SparkQA · 2019-01-22T16:25:08Z

Test build #101535 has finished for PR 23609 at commit 687c3e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-22T16:59:47Z

Test build #101537 has finished for PR 23609 at commit 687c3e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-23T19:05:35Z

Hi, @srowen . Could you review this please?

srowen · 2019-01-23T20:39:22Z

I honestly don't know anything about this part of the code. Eh, would @rdblue have a comment as it sorta concerns the v2 DataSource?

arunmahadevan · 2019-01-23T21:04:16Z

@HeartSaVioR , the changes looks good, but I am not very sure how the optimizer works. Did you check if the Literal could get pushed down ? If so this might cause issues like https://issues.apache.org/jira/browse/SPARK-18339 for which the CurrentBatchTimestamp was added earlier.

HeartSaVioR · 2019-01-23T21:47:18Z

@arunmahadevan That's why I don't transform actual plan and have temporal plan which is used just for extracting schema. Actual plan is not changed.

arunmahadevan · 2019-01-23T21:54:24Z

Makes sense, LGTM

rdblue · 2019-01-23T22:51:41Z

It sounds like this is related to DSv2 only because v2 gets the schema of a logical plan, which previously contained unresolved nodes. If the fix is to resolve the nodes, then this should be fine. If the fix were to change how v2 works and avoid some call to get the schema of a plan, then I would be concerned.

HeartSaVioR · 2019-01-23T22:55:52Z

If the fix is to resolve the nodes, then this should be fine. If the fix were to change how v2 works and avoid some call to get the schema of a plan, then I would be concerned.

This patch is to resolve the nodes prior to get a schema (via having temporary plan), while it doesn't touch the actual plan so that it would work as before.

Btw, I agree DSv2 is a moving target, but the bug is affecting not only for master branch but also for 2.4/2.3 as well. So I hope we get it resolved first, and fix it (again) when new DSv2 is concerning.

dongjoon-hyun · 2019-01-25T19:06:04Z

Retest this please.

dongjoon-hyun

+1, LGTM (pending Jenkins). This is a safe and clean fix to get only the schema.
Also, we had better have this test coverage during DSv2 transition.

HeartSaVioR · 2019-01-25T23:28:35Z

Thanks all for reviewing and thanks Dongjoon for merging!

HeartSaVioR · 2019-01-25T23:31:47Z

@dongjoon-hyun Given the bug affects 2.3 and 2.4 version lines as well, would we be better to have backport PRs as well?

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes #23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2019-01-25T23:34:42Z

It's merged to branch-2.4 already. It took some time for me to test manually on branch-2.4.

dongjoon-hyun · 2019-01-25T23:35:09Z

For branch-2.3, could you make a backport PR explicitly, @HeartSaVioR ? It would be more safer.

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

HeartSaVioR · 2019-01-25T23:40:26Z

Sure, I'll craft it quickly. Thanks @dongjoon-hyun!

dongjoon-hyun · 2019-01-26T01:40:38Z

Ur, @HeartSaVioR . The patch resolved the original reported issue, but the test case seems to be not enough to reproduce it properly. Could you check the test case once more?

HeartSaVioR · 2019-01-26T01:54:43Z

Ah that's odd. I'll see whether test case is valid again and fix if not. Thanks for reporting.

(Maybe later today or not today, given that I have some errands...)

dongjoon-hyun · 2019-01-26T06:39:11Z

Sorry, guys. I misanalyzed the root cause. I'll make a followup PR.

dongjoon-hyun · 2019-01-26T07:16:20Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala

+    val input = MemoryStream[Int]
+    val df = input.toDS()
+      .withColumn("cur_timestamp", lit(current_timestamp()))
+      .withColumn("cur_date", lit(current_date()))


Currently, this hides current_timestamp and makes this test case succeeds always.

dongjoon-hyun · 2019-01-26T07:17:21Z

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala

      case cd: CurrentDate =>
        CurrentBatchTimestamp(offsetSeqMetadata.batchTimestampMs,
          cd.dataType, cd.timeZoneId)
    }


The root cause is CurrentBatchTimestamp is TimeZoneAwareExpression which is unresolved without TimeZoneId.

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes #23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 1ca6b8b) Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 1ca6b8b) Signed-off-by: Dongjoon Hyun <[email protected]>

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 1ca6b8b) Signed-off-by: Dongjoon Hyun <[email protected]>

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 1ca6b8b) Signed-off-by: Dongjoon Hyun <[email protected]>

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache/spark#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes #23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 1ca6b8b) Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 448a063)

HeartSaVioR commented Jan 22, 2019

View reviewed changes

SPARK-26379 Fix issue on adding current_timestamp/current_date to str…

afb0dc4

…eaming query

HeartSaVioR force-pushed the SPARK-26379 branch from dc09c99 to afb0dc4 Compare January 22, 2019 08:24

Fix build issue which local maven wasn't caught as error strangely

8f30a75

gaborgsomogyi reviewed Jan 22, 2019

View reviewed changes

Address review comments from @gaborgsomogyi

d09845b

gaborgsomogyi approved these changes Jan 22, 2019

View reviewed changes

Reduce test time

687c3e4

dongjoon-hyun approved these changes Jan 25, 2019

View reviewed changes

dongjoon-hyun closed this in a4e4835 Jan 25, 2019

HeartSaVioR deleted the SPARK-26379 branch January 25, 2019 23:28

dongjoon-hyun mentioned this pull request Jan 26, 2019

[SPARK-26379][SS][BRANCH-2.3] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp #23656

Closed

dongjoon-hyun reviewed Jan 26, 2019

View reviewed changes

dongjoon-hyun mentioned this pull request Jan 26, 2019

[SPARK-26379][SS][FOLLOWUP] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp #23660

Closed

[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query #23609

[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query #23609

Uh oh!

Conversation

HeartSaVioR commented Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HeartSaVioR commented Jan 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

gaborgsomogyi commented Jan 22, 2019

Uh oh!

HeartSaVioR commented Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaborgsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborgsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

gaborgsomogyi commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

gaborgsomogyi commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 22, 2019

Uh oh!

dongjoon-hyun commented Jan 23, 2019

Uh oh!

srowen commented Jan 23, 2019

Uh oh!

arunmahadevan commented Jan 23, 2019

Uh oh!

HeartSaVioR commented Jan 23, 2019

Uh oh!

arunmahadevan commented Jan 23, 2019

Uh oh!

rdblue commented Jan 23, 2019

Uh oh!

HeartSaVioR commented Jan 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HeartSaVioR commented Jan 22, 2019 •

edited

Loading

HeartSaVioR commented Jan 22, 2019 •

edited

Loading

HeartSaVioR commented Jan 23, 2019 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Jan 25, 2019 •

edited

Loading

dongjoon-hyun commented Jan 25, 2019 •

edited

Loading

HeartSaVioR commented Jan 26, 2019 •

edited

Loading