File tree Expand file tree Collapse file tree 1 file changed +4
-5
lines changed
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming Expand file tree Collapse file tree 1 file changed +4
-5
lines changed Original file line number Diff line number Diff line change @@ -63,7 +63,7 @@ case object MinWatermark extends MultipleWatermarkPolicy {
6363}
6464
6565/**
66- * Policy to choose the *min * of the operator watermark values as the global watermark value. So the
66+ * Policy to choose the *max * of the operator watermark values as the global watermark value. So the
6767 * global watermark will advance if any of the individual operator watermarks has advanced.
6868 * In other words, in a streaming query with multiple input streams and watermarks defined on all
6969 * of them, the global watermark will advance as fast as the fastest input. So if there is watermark
@@ -108,10 +108,9 @@ case class WatermarkTracker(policy: MultipleWatermarkPolicy) extends Logging {
108108 }
109109 }
110110
111- // Update the global watermark to the minimum of all watermark nodes.
112- // This is the safest option, because only the global watermark is fault-tolerant. Making
113- // it the minimum of all individual watermarks guarantees it will never advance past where
114- // any individual watermark operator would be if it were in a plan by itself.
111+ // Update the global watermark accordingly to the chosen policy. To find all available policies
112+ // and their semantics, please check the comments of
113+ // `org.apache.spark.sql.execution.streaming.MultipleWatermarkPolicy` implementations.
115114 val chosenGlobalWatermark = policy.chooseGlobalWatermark(operatorToWatermarkMap.values.toSeq)
116115 if (chosenGlobalWatermark > globalWatermarkMs) {
117116 logInfo(s " Updating event-time watermark from $globalWatermarkMs to $chosenGlobalWatermark ms " )
You can’t perform that action at this time.
0 commit comments