[BUG] OTA API and OTA_EventProcessingTask is not task/thread safe when it comes to accessing common state.

[phelter]

Describe the bug
The OTA API and the task that is expected to be used use common data values without synchronization between tasks/threads. The OTA implementation is NOT Thread/Task safe.

There is a gross error in the way portions of the otaAgent internal state is being read/modified/written. Portions of it are assumed to be atomic across all tasks/threads but there are no guarantees that this is the case.

There are 3 Potential tasks/threads where actions can be performed and are currently in contention:

• Application task - executing the OTA_* api - eg: OTA_Shutdown(), OTA_GetState(), OTA_SignalEvent(), OTA_ActivateNewImage(), etc.
• OTA_EventProcessingTask()
• Network task (mqtt or http) executing the callbacks
• Timer task - for timers - This one is okay because all of the timers used are sending Events via a queue to the EventProcessingTask.

For the state and or callbacks there is no synchronization barrier (eg a semaphore or mutex) of the otaAgent information when any of these three tasks are accessing the otaAgent common control block.

These values MUST be either specified as atomic OR consumed within a semaphore/mutex lock so that actions performed upon them by either a task calling the OTA_*() API functions or the task running OTA_EventProcessingTask() will not inadvertently overwrite the values - especially within code portions that have - read - decision - write

I'm only providing the examples pertaining to the API (App -> OTA_EventProcessingTask()) but there are most likely others between the Network registered callbacks and the OTA_EventProcessingTask() as well.

Eg: the OTA_Init()

This should have something along the lines of:

    if (otaAgent.lock == NULL)
    {
        otaAgent.lock = xSemaphoreCreateMutex();
        assert(otaAgent.lock != NULL);
    }
    BaseType_t semRet = xSemaphoreTake(otaAgent.lock, portMAX_DELAY);
    assert(pdTRUE == semRet);
    (void)semRet;
    // All reads and/or modifications of otaAgent and it's associated values.
    //  Lines - https://github.com/aws/ota-for-aws-iot-embedded-sdk/blob/c3bd5840979cadfe1f9505e13e49cccb87333650/source/ota.c#L3264-L3347

    semRet            = xSemaphoreGive(otaAgent.lock);
    assert(pdTRUE == semRet);

Other API's that require this type of change are:

• OTA_Shutdown - requires local copy of state and then return outside of semaphore/mutex lock.
• OTA_GetState - requires local copy of state and then return outside of semaphore/mutex lock.
• OTA_GetStatistics - otherwise portions of the stats may not be correct relative to each other. - might suggest a separate lock for this.
• OTA_ActivateNewImage - requires creating a local copy of the ?? activateFn = otaAgent.pOtaInterface->pal.activate and then using that if not null.
• OTA_SetImageState - required when setImageStateWithReason() is used.
• OTA_GetImageState - requires creating a local copy of the imageState within a lock.
• OTA_Suspend - should move that code into the action performed by the OtaAgentEventSuspend message being received by the OTA_EventProcessingTask
• OTA_Resume - stopped here - you get the idea...
• OTA_SignalEvent - for the statisitcs and read of state - the stats should probably have their own lock

API that looks to be okay:

• OTA_CheckForUpdate()
• OTA_Err_strerror()
• OTA_JobParse_strerror
• OTA_PalStatus_strerror
• OTA_OsStatus_strerror

As mentioned, did not check any of the handlers that are registered to the network - but assuming there are most likely the same level of issue here.

Host

• Host OS: Linux - but this is ANY OS including FreeRTOS
• Version: Ubuntu 18.04

To Reproduce

• N/A - done by inspection, but Could reproduce by running this through Thread Sanitizer (clang) and discovering the errors.

Expected behavior

See Above - expected all API calls that use or modify otaAgent.* internal construct - which is used by other tasks, the access of those fields are protected by a semaphore and/or mutex.

Screenshots

N/A

Wireshark logs

N/A

Additional context

N/A

[shubnil]

Hi, We are looking into the same and will get back soon.

[kstribrnAmzn]

Thanks for this extremely detailed bug report @phelter. I spent a few hours digging through our code trying to prove thread safety to myself and I agree - its sorely lacking. The application task, processing loop, and callbacks could easily clash. The OTA library predates me a bit so I'd like to chat with a few people on my team who were here during its most recent iterations development to understand if I'm missing some information on these functions call patterns.

At this moment I'm in agreement with the expected behavior you've described.

[phelter]

@kstribrnAmzn - Thanks for acknowledging this issue.

Would it be possible to provide an estimate time for this to be corrected? So that I can plan accordingly.

Unfortunately, I don't believe this is the only FreeRTOS or Amazon IoT C-library to have this issue.
I've found the following Bug 570 in the FreeRTOS-Plus-TCP as well which is due to state not being protected across threads.

It might be prudent to look at and confirm the following don't have similar issues as well.

• coreMQTT
• coreMQTT-Agent
• Device-Shadow-for-AWS-IoT-embedded-sdk
• Device-Defender-for-AWS-IoT-embedded-sdk
• corePKCS11
• coreHTTP
• Jobs-for-AWS-IoT-embedded-sdk

Since the FreeRTOS/coreJSON and FreeRTOS/backoffAlgorithm don't have any task/thread based expectations, I believe they are okay.

[kstribrnAmzn]

Sorry for the delay in responding to you @phelter. Checking our other libraries makes a lot of sense. We want consistency across the FreeRTOS software suite. I'll create stories in our backlog queue so that we don't loose track of this work.

As for an estimate - I cannot provide one. Restructuring OTA is high on our priority list however things have shifted a bit recently. I'm not sure exactly when this work will be started and by how many. Any date I provide would be inaccurate and likely out of date within weeks.

[kstribrnAmzn]

I've created both issues on the Github repositories mentioned requesting help wanted (to get faster support from the community) as well as an internal tracking ticket for the FreeRTOS team.

[eldruid]

I am just new to devoloping IoT with AWS. is there anything I can assist with. volunteer work? please message me. thanks in advance. i look forward to contributing in any way. cheers 497115@protonmail.com

[kstribrnAmzn]

@eldruid Thanks for reaching out! Between all the repositories my team own (FreeRTOS), there is plenty of work to do.

What work or libraries interest you?

In the meantime I'm reaching out to others on my team to see if we have any work which would be doable for someone not as familiar with the libraries.This issue you commented on could be a good place to start. There is essentially 2 parts to the work as of right now.

The first part is auditing our various libraries, as @phelter had mentioned, to ensure there are no other variables which can be accessed in a thread unsafe manner. Basically, you'd want to look for structures/objects which contain library state and are used by both the library and optionally the user. The issue with these is that the user could be accessing the structure/object from another FreeRTOS task (aka thread) while the library task/thread is updating the values, causing a race condition.

The second part of the work is implementing a mutex preventing the race condition. In the case of this issue, the mutex would block access in the case that the object is being updated. In our case, and since this library is not tied to a specific OS, may require further rework. A set of getter methods for the object which requires the OTA task to synchronize could be a better solution. I will say I haven't thought deeply about this so I'm not sure either of these are the right solutions.