Pod not pairing

[kwbeekman]

Markdown content.zip

Describe the bug
My 16 yo tried to pair a new pod on 6-19 around 3 pm. Error message that RL couldn’t identify pod. She was in her car with RL about 2 inches from pod in her lap. She tried several times but didn’t know to “delete” the pod. She then tried to deactivate it and the pod started screaming. She used the pin in the hole technique to turn it off, waited until she got home and tried a second pod, but got the same message. Eventually, I got home. Drove the first pod 5 mi away. Followed directions on loopdocs and got the “ack” message. Could have been by this time it had been a few hrs since she had first tried the 2nd pod. I. Tried deleting/rebuilding app, multiple turn offs. At one point, now midnight outside our house, shut everything down and restarted. Never got it to work. Next day tried a new pod and went smoothly.

Attach an Issue Report
I wasn’t sure if I needed to edit something out of this so am just attaching entire thing.

To Reproduce
Not sure I could.

Phone

• Iphone 11
• OS Version: 13.5.1

Loop Version

• Version Number: 2.2

CGM

• Device: Dexcom G6
• Manager app: dexcom

Pump

• Manufacturer: Omnipod
• Model: Eros
• Firmware version: ? Not sure

**Additional Loading...

@#`

`

[Kdisimone]

There's no Issue Report attached here. If you got one, please upload it to this thread. Without an issue report, there's nothing much we can look at to debug. https://loopkit.github.io/loopdocs/troubleshooting/overview/#issue-report

[ps2]

It's attached at the top (the "Markdown content" zip file)

[Kdisimone]

Ah thanks @ps2. I missed that.

[marionbarker]

@ps2 - I have seen this pattern several times where 0x07, 0x0115 are ok, 0x03 is sent followed by an ACK from pod, and then Loop moves on to the 0x08 command; and then the user has trouble getting pod to pair.
@itsmojo told me earlier that sometimes it's ok to move on with an ACK after 0x03 but sometimes not. I can post (or send via zulipchat) more parsed information on this report and more reports if you'd find it helpful. (Some reports I picked up from Facebook and asked folks to PM me with the reports. I was mostly focused on getting them paired and looping again.) I got the impression you didn't want to make rileylink_ios mods until after the next release, but if you want the information now, I'll try to get it organized. (Note: actPP column below contains the last pod progress state detected - it didn't update after the 0x0115 response was detected).

  Report on Omnipod from kwbeekman/LoopReport_2020-06-19 23-05-39.md
     Block 2 of 4

Pod never reached pod_progress of 8
    *** pod exceeded nominal init steps of 18 ***
    Pod: Address 0x1f01159d, Lot 45269, PI: 2.9.0, gain 1, rssi 61, numInitSteps 45

  CumSec: seqNum: expectAction  : expMT  :success: actMT  : actPP: ppMeaning
   1835:       0: assignID      : 0x07   :    y  : 0x07   :     0:               
   1842:       1: successID     : 0x0115 :    y  : 0x0115 :     2: ReminderInitialized
   1843:       2: setupPod      : 0x03   :    y  : 0x03   :     2: ReminderInitialized
   1874:       3: successSetup  : 0x011b :       : 0x03   :     2: ReminderInitialized
   1877:       1: successSetup  : 0x011b :       : ACK    :     2: ReminderInitialized
   1877:       4: successSetup  : 0x011b :       : 0x08   :     2: ReminderInitialized
   1893:       5: successSetup  : 0x011b :       : 0x08   :     2: ReminderInitialized
   1931:       6: successSetup  : 0x011b :       : 0x08   :     2: ReminderInitialized

[ps2]

Thanks for the report, @kwbeekman. Looking into it now.

Here are the relevant portions of the logs, where pairing failed: https://gist.github.com/ps2/1929ad4a0077a8793271fd1074fef0b8

[itsmojo]

There were a number of different problems which occurred in this three pod failure sequence by Kathleen Beekman. These failures all seemed to be caused by a RileyLink placement too close to the pod causing some pod responses not to be reliably heard by Loop which led to a number of different pairing issues.

In the first pod failure, the 1st SetupPod command at 18:56:23 received no response and the 2nd SetupPod command attempted 31 seconds later on the next Pair attempt at 18:56:54 received an ack and then all the subsequent FaultConfig commands apparently received no responses. What actually happens in these cases is that the pod will end up using a packet address of 0xFFFFFFFF for the FaultConfig response and any subsequent responses throughs the pod's lifetime due to the way the firmware and the pairing process works. This results in all the pod's responses being dropped by Loop which currently doesn't accept the packets with a packet address of 0xFFFFFFFF after pairing while the S08 MCU in the PDM will. Thus it appears that the pod is not responding when in fact it is (albeit with a misconfigured packet address). This failure can be reliably reproduced in Loop by setting up an artificial situation where non-ack packets for the 2-packet 011B response for the SetupPod command are dropped (or if the Loop ack's to acknowledge receipt of either the first or second packet in the 2-packet 011B response are never sent) and there is about 25 seconds or more before the second user Pair is attempted. Landing pending rileylink_ios PR's ps2/rileylink_ios#597 & ps2/rileylink_ios#598 would have correctly handled this pod and all the other pods lost to apparent non-response issues after receiving an ack from an SetupPod command.

In the second pod failure, the VersionResponse at 21:52:03 for the AssignAddress had a podProgressStatus value of .activationTimeExceeded (=14). A pod enters this progress state when an #7 alert timer expires during pairing (either 2 hours from power up to prime start or 1 hour from prime start to pod ready). In this case, this progress status value indicates that 2 hours had passed since the pod was filled with insulin. Since a pod fault is not returned for the AssignAddress or SetupPod commands for an Activation time exceeded condition, the podProgress value in the VersionResponse's needs to be explicitly checked in the pairing code for this condition. The SetupPod VersionResponse handling has a guard statement which verifies that the podProgress is .podConfigured (=3), so the 2nd VersionResponse at 21:52:05 is rejected with an invalidData error. When the user tries to Pair again after getting the somewhat bogus invalid data error, the second SetupPod command sent at 21:52:20 will receive an ack (because the SetupPod command has actually successfully run) and then Loop will move on to attempt a FaultConfig command which receives an confusing "No fault" pod fault response (a fault with a 0 error code) due to the activation time exceeded condition with a non 07 or 03 command. Since this particular "No fault" pod fault can also occur anytime after the two step pairing commands during priming, basal setup & cannula insertion, to correctly handle "Activation time exceeded" failures all of the Loop fault checking code needs to be upgraded to handle a pod progress value of .activationTimeExceeded conditions most anywhere to correctly handle this as an alternate type of pod fault. Among other things this means that a PodState might need to be created just to save the state that a pod has faulted due to an activation timeout error and pod faults need to look at the podProgress to determine if the fault code is actually meaningful.

In the third pod failure, repeated ack's are being returned for AssignAddress commands. This condition occurs whenever AssignAddress commands with fixed packet #'s are being heard by the pod, but the pod's VersionResponses are not being heard by Loop for an extended period. Once this happens subsequent packets sent to the pod using the same packet # will generate an ack (with the packet seq # one more than the AssignAddress packet #). This failure can be reliably reproduced in Loop by setting up an artificial situation where all the non-ack response packets for the AssignAddress command are dropped (or if the Loop ack to acknowledge the one packet 0115 response was received is not sent) and there is about 25 seconds or more before the second user Pair is attempted. When the PDM pairs to a non-responsive/non-existent pod, it will continue to attempt to use packet #0 and the same pod address for an AssignAddress command for about 20 seconds, printing the "Communications error" message with beeping requesting repositioning about halfway through. On each subsequent pairing attempt, the PDM will use the same pod address for the AssignAddress, but will increment the packet # by 1 each time to prevent the pod from getting into a constant AssignAddress ack condition.

All of these pairing failures in these examples occurred because the RileyLink was apparently being placed too close to the pods as indicated by the repeated rssi values of 61 when there was an AssignAddress response. When the rssi value is too high the pod responses via the RileyLink are sometimes not successfully received by Loop, but of course this also occurs when the rssi value is too low. Fixing the 07 ack issue and the transport issue for pods using a packet address of 0xFFFFFFFF should be able to basically completely eliminate the biggest remaining pairing problems. But the general pairing problem is when communications are not reliable during the 03 SetupPod & the subsequent command is that there is always a risk of a pod ending up running misconfigured with packets #'s of 0xFFFFFFFF throughout its lifetime. Since the 07 AssignAddress command is quite retriable (at least when the packet # is not always fixed to be 0 ;-)) and its response includes an rssi value which is not present in the longer 03 SetupPod response (which has fixed padding bytes to force use of a continuation packet), it would seem that the Omnipod pairing might have been designed to have the PDM use the returned AssignAddress rssi value to determine if the pod is properly positioned before attempting pairing. The PDM doesn't have to consider the case of a pod being too close to the PDM as the recommended position for pairing is with the pod touching the side of the PDM since the PDM uses an S08 MCU instead of the stronger RL hardware for its radio which apparently is contributing to the problems for too close of a placement. Upon looking at some recent pairing results as tabulated by Marion Barker, I found that the number of pods which were lost because of a failure to properly configure its radio ID (i.e., apparently had no response to 08 FaultConfig commands after an SetupPod ack) was 17% with an rssi in the 10's, 6% with an rssi in the 20's, 6% with an rssi in 60's and < 1% with an rssi value in the 30's, 40's and 50's. It would seem that having Loop verify that the rssi value is not too low or too high before continuing with the pairing would be a prudent & helpful thing to do which would highly reduce the probability of a pod ending up having to run with a misconfigured address of 0xFFFFFFFF or creating other potential pairing difficulties due to pod responses not being heard reliably by Loop during pairing.

[itsmojo]

Here's some logs showing an 03 ACK leading to an apparent non-responsive pod which is then recovered from after enabling the proposed fix.
Example_Loop_messages_for_apparent_nonresponsive_pod_and_recovery_after_03_ACK.txt
Example_Parser_output_for_apparent_nonresponsive_pod_and_recovery_after_03_ACK.txt

Here's a log showing a created constant 07 ACK situation which is then recovered from after enabling the proposed fix. This log was taken from a more detailed testing session which is also attached a number of other related situations that can occur.
Example_AssignAddress_Constant_ACK_Failure_and_Recovery.txt
Detailed_AssignAddress_ACK_testing.txt