Skip to content

Commit aac7be0

Browse files
committed
Added information on how to get partition id
1 parent a66ec22 commit aac7be0

File tree

1 file changed

+10
-2
lines changed

1 file changed

+10
-2
lines changed

docs/streaming-programming-guide.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2218,8 +2218,16 @@ additional effort may be necessary to achieve exactly-once semantics. There are
22182218

22192219
- *Transactional updates*: All updates are made transactionally so that updates are made exactly once atomically. One way to do this would be the following.
22202220

2221-
- Use the batch time (available in `foreachRDD`) and the partition index of the transformed RDD to create an identifier. This identifier uniquely identifies a blob data in the streaming application.
2222-
- Update external system with this blob transactionally (that is, exactly once, atomically) using the identifier. That is, if the identifier is not already committed, commit the partition data and the identifier atomically. Else if this was already committed, skip the update.
2221+
- Use the batch time (available in `foreachRDD`) and the partition index of the RDD to create an identifier. This identifier uniquely identifies a blob data in the streaming application.
2222+
- Update external system with this blob transactionally (that is, exactly once, atomically) using the identifier. That is, if the identifier is not already committed, commit the partition data and the identifier atomically. Else if this was already committed, skip the update.
2223+
2224+
dstream.foreachRDD { (rdd, time) =>
2225+
rdd.foreachPartition { partitionIterator =>
2226+
val partitionId = TaskContext.get.partitionId()
2227+
val uniqueId = generateUniqueId(time.milliseconds, partitionId)
2228+
// use this uniqueId to transactionally commit the data in partitionIterator
2229+
}
2230+
}
22232231

22242232

22252233
***************************************************************************************************

0 commit comments

Comments
 (0)