FAILED: CRDTActorOwnedTests.test_actorOwned_GCounter_increment_shouldResetDelta_shouldNotifyOthers

[ktoso]

00:02:46.205 Test Case 'CRDTActorOwnedTests.test_actorOwned_GCounter_increment_shouldResetDelta_shouldNotifyOthers' started at 2019-09-24 16:19:34.447
00:02:46.207 2019-09-24T16:19:34+0000 info: deadLetter=1 [CRDTActorOwnedTests][EventStream.swift:41][/dead/letters] Dead letter: [subscribe(ActorRef<ClusterEvent>(/system/replicator/$sub-DistributedActors.ClusterEvent-y))]:DistributedActors.EventStreamShell.Message<DistributedActors.ClusterEvent> was not delivered to ["/dead/letters"].
00:02:49.214 <EXPR>:0: error: CRDTActorOwnedTests.test_actorOwned_GCounter_increment_shouldResetDelta_shouldNotifyOthers : threw error "error(message: "\n        try g1Owner2EventP.expectMessage(.ownerDefinedOnUpdate)\n                            \u{1B}[0;31m^~~~~~~~~~~~~~\nerror: Did not receive expected [ownerDefinedOnUpdate]:OwnerEventProbeMessage within [3s], error: noMessagesInQueue\u{1B}[0;0m")"
00:02:49.216 Test Case 'CRDTActorOwnedTests.test_actorOwned_GCounter_increment_shouldResetDelta_shouldNotifyOthers' failed (3.011 seconds)

FYI @yim-lee

https://ci.swiftserver.group/job/swift-distributed-actors-swift50-p1rb/231/console

[yim-lee]

Hmm, looks like a re-occurrence of #14?

[yim-lee]

This is the failed assertion: https://github.com/apple/swift-distributed-actors/blob/master/Tests/DistributedActorsTests/CRDT/CRDTActorOwnedTests.swift#L169

To summarize what the test does:

• GCounter g1 is supposed to have two registered owners: o1 and o2.
• o1 updates g1.
• Replicator should notify both o1 and o2.

The assertions fails because o2 didn't receive the notification. o1 did.

The messages involved are LocalCommand.register (both o1 and o2) and LocalCommand.write (o1 only).

We "fixed" the previous issue (#14) by using tell instead of ask for LocalCommand.register. LocalCommand.write (and others) still uses ask.

Since o1 received notification, I think it's safe to say LocalCommand.write worked fine.

And that leaves only LocalCommand.register. It's highly likely that o2 didn't register with the replicator and therefore didn't receive the notification. Don't know the cause though.

[ktoso]

Thanks for the analysis!
I think ask this time around is unlikely to be the culprit... we fixed it in #33

I'll have a look when I have a few cycles to spare today

[yim-lee]

Thanks! Let me know if I can help. Curious if it was caused by the status quo at the time or something more ominous.

[ktoso]

Note to self, look at this today

[ktoso]

Likely similar/same cause: #172