tests/kernel/timer/timer_api fails on nrf52_pca10040 board

[spoorthik]

Describe the bug
The CI run caught test_timer_periodicity test case failure in tests/kernel/timer/timer_api on nrf52_pca10040 board. This issue is sporadic and I couldn't reproduce locally.

To Reproduce
Steps to reproduce the behavior:

1. mkdir build; cd build
2. cmake -DBOARD= nrf52_pca10040
3. make flash
4. See error

Expected behavior
All test cases should pass without any assertion failure or crash.

Screenshots or console output

********** delaying boot 1000ms (per build configuration) *****
***** Booting Zephyr OS zephyr-v1.13.0-2205-g8d84f9229c (delayed boot 1000ms) *****
Running test suite timer_api
===================================================================
starting test - test_timer_duration_period
PASS - test_timer_duration_period
===================================================================
starting test - test_timer_period_0
PASS - test_timer_period_0
===================================================================
starting test - test_timer_expirefn_null
PASS - test_timer_expirefn_null
===================================================================
starting test - test_timer_periodicity

    Assertion failed at /home/jenkins/workspace/zephyr-master-ci-master/zephyr.git/tests/kernel/timer/timer_api/src/main.c:240: test_timer_periodicity: (((delta) >= (50)) && ((delta) <= (50) + (1))) is false

FAIL - test_timer_periodicity
===================================================================
starting test - test_timer_status_get
PASS - test_timer_status_get
===================================================================
starting test - test_timer_status_get_anytime
PASS - test_timer_status_get_anytime
===================================================================
starting test - test_timer_status_sync
PASS - test_timer_status_sync
===================================================================
starting test - test_timer_k_define
PASS - test_timer_k_define
===================================================================
starting test - test_timer_user_data
PASS - test_timer_user_data
===================================================================
Test suite timer_api failed.
===================================================================
PROJECT EXECUTION FAILED

Environment (please complete the following information):

• OS: Linux
• Toolchain: Zephyr SDK 0.9.5

Since the failure is not seen locally, I couldn't git bisect to find on what commit the failure was first seen.

[carlescufi]

@pizi-nordic can you take a look?

[pizi-nordic]

Interesting... We are calling _add_timeout(&timeout, handler, 5), but the handler is fired 4 ticks later.

[pizi-nordic]

starting test - test_timer_periodicity                                                                                                                                                      
PERIOD: 50, DELTA: 40                                                                                                                                                           
                                                                                                                                                                                              
    Assertion failed at /home/pizi/repos/zephyr/tests/kernel/timer/timer_api/src/main.c:250: test_timer_periodicity: (((delta) >= (50)) && ((delta) <= (50) + (1))) is false                
                                                                                                                                                                                              
FAIL - test_timer_periodicity                                                                                                                                                               
[00:00:00.827,880] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 1                                                                                                                        
[00:00:00.827,911] <inf> timer._impl_k_timer_start: new timeout 0x200000E0 (will expire in 0 ticks)                                                                                         
[00:00:00.827,911] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 0  <= This moves the announce on the next tick boundary.                                                                                                                       
[00:00:00.827,941] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 0  <= This moves the announce on the next tick boundary.                                                                                                                      
[00:00:00.838,256] <inf> timeout.z_clock_announce: ticks: 2 <= Two ticks to process.                                                                                                                                
[00:00:00.838,256] <inf> timeout.z_clock_announce: timeout 0x200000E0 expired! <= First of the 2 ticks announced triggers timer.                                                                                                             
[00:00:00.838,256] <inf> timer._timer_expiration_handler: new timeout 0x200000E0 (will expire in 5 ticks) <= New timeout set.                                                                                                                                                                                                         
[00:00:00.838,256] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 5 <= HW correctly programmed.
[00:00:00.838,287] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 0                                                                                                                        
[00:00:00.838,287] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 4 <= Timeout is adjusted when the second of the 2 ticks announced is processed (due to t->dticks -= announce_remaining)                                                                                                    
[00:00:00.838,317] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 0                                                                                                                        
[00:00:00.848,236] <inf> timeout.z_clock_announce: ticks: 1                                                                                                                                 
[00:00:00.848,236] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 3                                                                                                                        
[00:00:00.868,194] <inf> timeout.z_clock_announce: ticks: 2                                                                                                                                 
[00:00:00.868,194] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 1                                                                                                                        
[00:00:00.878,173] <inf> timeout.z_clock_announce: ticks: 1  <= 4th tick announced.                                                                                                                               
[00:00:00.878,173] <inf> timeout.z_clock_announce: timeout 0x200000E0 expired! <= Timeout fired too early.                                                                                                          
[00:00:00.878,173] <inf> nrf_rtc_timer.z_clock_set_timeout: ticks: 5  <= Start of next timer period....                                                                                                                      
[00:00:00.878,173] <inf> timer._timer_expiration_handler: new timeout 0x200000E0 (will expire in 5 ticks)                                                                                   

@andyross:
It looks that logic around curr_tick in z_clock_announce() is not working well together with announcement of more than one tick. As result we are adjusting dticks of timeouts which were added in the previous loop iteration. IMHO we should first move all the timeouts which are supposed to be executed to a temporary list, and then just run through it.

[andyross]

Hm. That code is by design, actually. The intent is indeed that (for an example) if you have a callback that is supposed to have fired at timer T (but is actually being handled at T+3 because the interrupt arrived late), and it sets a timeout of its own to fire in 1 tick, that that tick gets handled synchronously inside the same ISR at wherever "T+2" timeouts are being issued. Doing it in a separate list would defeat that.

Will take a look. I have a 10056 board, not 10040, though it seems like they're behaving identically?

[pizi-nordic]

Yes, both 10040 and 10056 uses the same timer.

[pdunaj]

Hi @andyross , this behavior may trigger serious problems as module scheduling a timeout may require separation of T between the callbacks (e.g. because of HW constraints when you sample a sensor). When scheduling a delay you expect that your callback will be fired no sooner then T (how accurately it happens depends on how close to real time OS the system is).
The code here will not only call the first callback with a delay of 3T. It will also immediately call it second and third time without maintaining the constraint not sooner then. Module using it will not be aware of the delay at all but if you communicate with the real world this delay will be obviously seen (imagine sensor sampling).

The problem I see is that even if the callback is called with a delay, it will not know about it as curr_tick will not hold the real passing of time (as it is being updated with each cycle loop). The solution may be to update the curr_tick or actually make z_tick_get return the true information about time, before we call any callback. This way callback would have a mean to check that they are delayed.

I don't think timeout system should hide from the other modules a fact that delay happened.

Collecting expired timeouts to the local list and fireing them after the loop is done would ensure that delay is not hidden as curr_tick is updated.

[pizi-nordic]

I think we have to choose between two modes of the operation:

• Current approach, when we are trying to compensate delayed firing of the timers:

  Expected timer firing: |--5--|--5--|--5--|---> (t)
  Actual timer firing:   |--5--|---7---|-3-|---> (t)
  

  In this approach we are trying to meet expectations about number of timer firings in given time interval. However this makes system time not monotinic, and in extreme case we can timer scheduled over short period of time could file almost immediately (timer scheduled in first iteration of loop in z_clock_announce() will fire in subsequent interation(s) if it was scheduled over shorter time than the ticks parameter passed to the announcement functions).

• Approach proposed in the previous comment, which makes system time monotonic, but accumulates error:

  Expected timer firing: |--5--|--5--|--5--|----> (t)
  Actual timer firing:   |--5--|---7---|--5--|--> (t)
  

  Here every period we are meeting the expectations (timer fires 5 ticks after scheduling or later), but error accumulates and as result timer could fire at lower rate than expected.

Also this bug shows me, that the "MIN_DELAY" concept in the RTC timer is not working perfectly, as every call to z_clock_set_timeout(0) could move the firing to the next tick boundary, introducing announcement of multiple ticks, which exposes the problem. We should not adjust the firing point if we know that the interrupt will fire in less than for example 1/2 of tick time.