Preventing torn reads with an HCS12 microcontroller

Multi tool use
Preventing torn reads with an HCS12 microcontroller
I'm trying to write an embedded application for an MC9S12VR microcontroller. This is a 16-bit microcontroller but some of the values I deal with are 32 bits wide and while debugging I've captured some anomalous values that seem to be due to torn reads.
I'm writing the firmware for this micro in C89 and running it through the Freescale HC12 compiler, and I'm wondering if anyone has any suggestions on how to prevent them on this particular microcontroller assuming that this is the case.
Part of my application involves driving a motor and estimating its position and speed based on pulses generated by an encoder (a pulse is generated on every full rotation of the motor).
For this to work, I need to configure one of the MCU timers so that I can track the time elapsed between pulses. However, the timer has a clock rate of 3 MHz (after prescaling) and the timer counter register is only 16-bit, so the counter overflows every ~22ms. To compensate, I set up an interrupt handler that fires on a timer counter overflow, and this increments an "overflow" variable by 1:
// TEMP
static volatile unsigned long _timerOverflowsNoReset;
// ...
#ifndef __INTELLISENSE__
__interrupt VectorNumber_Vtimovf
#endif
void timovf_isr(void)
{
// Clear the interrupt.
TFLG2_TOF = 1;
// TEMP
_timerOverflowsNoReset++;
// ...
}
I can then work out the current time from this:
// TEMP
unsigned long MOTOR_GetCurrentTime(void)
{
const unsigned long ticksPerCycle = 0xFFFF;
const unsigned long ticksPerMicrosecond = 3; // 24 MHZ / 8 (prescaler)
const unsigned long ticks = _timerOverflowsNoReset * ticksPerCycle + TCNT;
const unsigned long microseconds = ticks / ticksPerMicrosecond;
return microseconds;
}
In main.c
, I've temporarily written some debugging code that drives the motor in one direction and then takes "snapshots" of various data at regular intervals:
main.c
// Test
for (iter = 0; iter < 10; iter++)
{
nextWait += SECONDS(secondsPerIteration);
while ((_test2Snapshots[iter].elapsed = MOTOR_GetCurrentTime() - startTime) < nextWait);
_test2Snapshots[iter].position = MOTOR_GetCount();
_test2Snapshots[iter].phase = MOTOR_GetPhase();
_test2Snapshots[iter].time = MOTOR_GetCurrentTime() - startTime;
// ...
In this test I'm reading MOTOR_GetCurrentTime()
in two places very close together in code and assign them to properties of a globally available struct.
MOTOR_GetCurrentTime()
In almost every case, I find that the first value read is a few microseconds beyond the point the while loop should terminate, and the second read is a few microseconds after that - this is expected. However, occasionally I find the first read is significantly higher than the point the while loop should terminate at, and then the second read is less than the first value (as well as the termination value).
The screenshot below gives an example of this. It took about 20 repeats of the test before I was able to reproduce it. In the code, <snapshot>.elapsed
is written to before <snapshot>.time
so I expect it to have a slightly smaller value:
<snapshot>.elapsed
<snapshot>.time
For snapshot[8]
, my application first reads 20010014
(over 10ms beyond where it should have terminated the busy-loop) and then reads 19988209
. As I mentioned above, an overflow occurs every 22ms - specifically, a difference in _timerOverflowsNoReset
of one unit will produce a difference of 65535 / 3
in the calculated microsecond value. If we account for this:
snapshot[8]
20010014
19988209
_timerOverflowsNoReset
65535 / 3
A difference of 40 isn't that far off the discrepancy I see between my other pairs of reads (~23/24), so my guess is that there's some kind of tear going on involving an off-by-one read of _timerOverflowsNoReset
. As in while busy-looping, it will perform one call to MOTOR_GetCurrentTime()
that erroneously sees _timerOverflowsNoReset
as one greater than it actually is, causing the loop to end early, and then on the next read after that it sees the correct value again.
_timerOverflowsNoReset
MOTOR_GetCurrentTime()
_timerOverflowsNoReset
I have other problems with my application that I'm having trouble pinning down, and I'm hoping that if I resolve this, it might resolve these other problems as well if they share a similar cause.
Edit: Among other changes, I've changed _timerOverflowsNoReset
and some other globals from 32-bit unsigned to 16-bit unsigned in the implementation I now have.
_timerOverflowsNoReset
On another issue: Given that
ticksPerCycle
is 0xFFFF, you could save some overhead; by const unsigned long ticks = (_timerOverflowsNoReset << 16) + TCNT;
and if you could set the frequency to 1, 2 or 4MHz rather the 3, you could avoid the potentially really expensive division conversion for microseconds.– Clifford
Jul 2 at 18:41
ticksPerCycle
const unsigned long ticks = (_timerOverflowsNoReset << 16) + TCNT;
@Clifford I used
ticksPerCycle
because it's clearer than << 16
(it avoids magic numbers). However, I realised afterwards that it's an off-by-one error anyway (should be 0x10000
) and I've switched to << 16
. I assume that the compiler is smart enough to simplify multiplication to a shift operation when it's a power of 2. As for the part about microseconds, I'm only doing that temporarily for debugging purposes (easier to write tests in terms of microseconds than ticks) and otherwise I'd just be keeping all values in ticks.– Tagc
Jul 2 at 18:45
ticksPerCycle
<< 16
0x10000
<< 16
That is probably a poor reason - to write inefficient code because you cannot think of a better way to conform to a coding standard. instead use
sizeof(TCNT) * CHAR_BIT
or just define a constant such as counterBitLength = 16
and shift it instead of multiply. On the other hand a good compiler will fix it for you.– Clifford
Jul 2 at 18:55
sizeof(TCNT) * CHAR_BIT
counterBitLength = 16
I actually find a lot of embedded code to be pretty hard to read relative to code written in other languages, to the point where people even do things that have no impact on code performance whatsoever like abbreviating terminology (e.g. "timer" -> "tmr", "overflow" -> "ovf"). However, I'd have assumed that something
const unsigned long ticksPerCycle = 0x10000; something * ticksPerCycle
to compile to the same machine code as something << 16
, while being a lot more explicit to the reader what I'm doing.– Tagc
Jul 2 at 19:00
const unsigned long ticksPerCycle = 0x10000; something * ticksPerCycle
something << 16
7 Answers
7
Calculate the tick count, then check if while doing that the overflow changed, and if so repeat;
#define TCNT_BITS 16 ; // TCNT register width
uint32_t MOTOR_GetCurrentTicks(void)
{
uint32_t ticks = 0 ;
uint32_t overflow_count = 0;
do
{
overflow_count = _timerOverflowsNoReset ;
ticks = (overflow_count << TCNT_BITS) | TCNT;
}
while( overflow_count != _timerOverflowsNoReset ) ;
return ticks ;
}
the while loop will iterate either once or twice no more.
Thanks, this looks good. I repeated my test about 20 times with this implementation and I didn't observe any tearing, so I'm confident enough to use this. It's simpler than my solution and avoids the bug in it.
– Tagc
Jul 3 at 7:51
Regarding our earlier discussion, the one change I did make is to use
ticksPerCycle
instead of an explicit shift. hastebin.com/dedibaxofo.c I consider this more readable, and also more maintainable in case I later switch to using a dedicated timer channel match interrupt instead of a timer overflow interrupt (I just change the value of ticksPerCycle
). As I was hoping, the compiler produces identical machine code if you multiply by a power of 2, so there's no impact on performance. hastebin.com/bodemixoti.go– Tagc
Jul 3 at 7:52
ticksPerCycle
ticksPerCycle
@Tagc : That is fine, but using a full register overflow to synthesise a 32 bit timer from 16 bit hardware is not an uncommon solution. I would recommend your solution where an interrupting timebase is required in addition to a free-running counter, and you are constrained by hardware timer availability. Ideally I'd use a separate timer for that.
– Clifford
Jul 3 at 8:17
You can read this value TWICE:
unsigned long GetTmrOverflowNo()
{
unsigned long ovfl1, ovfl2;
do {
ovfl1 = _timerOverflowsNoReset;
ovfl2 = _timerOverflowsNoReset;
} while (ovfl1 != ovfl2);
return ovfl1;
}
unsigned long MOTOR_GetCurrentTime(void)
{
const unsigned long ticksPerCycle = 0xFFFF;
const unsigned long ticksPerMicrosecond = 3; // 24 MHZ / 8 (prescaler)
const unsigned long ticks = GetTmrOverflowNo() * ticksPerCycle + TCNT;
const unsigned long microseconds = ticks / ticksPerMicrosecond;
return microseconds;
}
If _timerOverflowsNoReset
increments much slower then execution of GetTmrOverflowNo()
, in worst case inner loop runs only two times. In most cases ovfl1
and ovfl2
will be equal after first run of while() loop.
_timerOverflowsNoReset
GetTmrOverflowNo()
ovfl1
ovfl2
This is a good way to do it. Other option would be to temporarily disable interrupts while reading the timer value but that introduces jitter.
– teroi
Jul 2 at 9:47
Thanks, this looks promising. I'll give this a try when I can.
– Tagc
Jul 2 at 10:01
Sorry but this is nonsense given that HCS12 is a limited 16 bit MCU which cannot read 32 bit variables like this. You'll rather end up reading the value 4 times, in very unsafe and uncontrolled ways - which is the problem inherited from the originally buggy code. Just don't do this, it won't solve anything.
– Lundin
Jul 2 at 11:06
This fails, when
GetTmrOverflowNo()
returns a stable value, but then TCNT overflows– jeb
Jul 2 at 11:38
GetTmrOverflowNo()
@teroi Disabling interrupts doesn't help in this case, as the overflow occours independ of interrupts.
– jeb
Jul 2 at 11:39
Based on the answers @AlexeyEsaulenko and @jeb provided, I gained understanding into the cause of this problem and how I could tackle it. As both their answers were helpful and the solution I currently have is sort of a mixture of the two, I can't decide which of the two answers to accept, so instead I'll upvote both answers and keep this question open.
This is how I now implement MOTOR_GetCurrentTime
:
MOTOR_GetCurrentTime
unsigned long MOTOR_GetCurrentTime(void)
{
const unsigned long ticksPerMicrosecond = 3; // 24 MHZ / 8 (prescaler)
unsigned int countA;
unsigned int countB;
unsigned int timerOverflowsA;
unsigned int timerOverflowsB;
unsigned long ticks;
unsigned long microseconds;
// Loops until TCNT and the timer overflow count can be reliably determined.
do
{
timerOverflowsA = _timerOverflowsNoReset;
countA = TCNT;
timerOverflowsB = _timerOverflowsNoReset;
countB = TCNT;
} while (timerOverflowsA != timerOverflowsB || countA >= countB);
ticks = ((unsigned long)timerOverflowsA << 16) + countA;
microseconds = ticks / ticksPerMicrosecond;
return microseconds;
}
This function might not be as efficient as other proposed answers, but it gives me confidence that it will avoid some of the pitfalls that have been brought to light. It works by repeatedly reading both the timer overflow count and TCNT register twice, and only exiting the loop when the following two conditions are satisfied:
This basically means that if MOTOR_GetCurrentTime
is called around the time that a timer overflow occurs, we wait until we've safely moved on to the next cycle, indicated by the second TCNT read being greater than the first (e.g. 0x0001 > 0x0000).
MOTOR_GetCurrentTime
This does mean that the function blocks until TCNT increments at least once, but since that occurs every 333 nanoseconds I don't see it being problematic.
I've tried running my test 20 times in a row and haven't noticed any tearing, so I believe this works. I'll continue to test and update this answer if I'm wrong and the issue persists.
Edit: As Vroomfondel points out in the comments below, the check I do involving countA
and countB
also incidentally works for me and can potentially cause the loop to repeat indefinitely if _timerOverflowsNoReset
is read fast enough. I'll update this answer when I've come up with something to address this.
countA
countB
_timerOverflowsNoReset
This code will crash and burn when an interrupt fires while the function is executed, because you still have the rookie bug of not protecting the variable shared with the ISR. This means you will eventually read half the variable from the previous timer cycle and the other half from the current one, leading to garbage results. You still need to do one of these 3, pick one: 1) either disable the interrupt while accessing the variable, 2) use some manner of semaphore or 3) ensure atomic access.
– Lundin
Jul 2 at 13:13
Btw you will keep accumulating your oscillator inaccuracy through algorithms like this. So depending on for how long you need to log values, you need to pick an oscillator accordingly. Internal RC oscillator is a no-go, but luckily most S12 don't have one (just in "limp home mode").
– Lundin
Jul 2 at 13:19
I would protect the variable if I could, but with this system I'm not sure how. I'm not using an RTOS like with other MCUs I've worked with, so I guess I'd have to do something like manually creating memory barriers in assembly. Maybe a volatile bool would work? Also, wouldn't disabling/enabling interrupts during calls to this function cause problems if I'm calling it inside loops that run thousands of times a second? I'm also not clear on what specific sequence of events could cause the crash/burn you describe - I thought double-reading the values would prevent that, even if tearing occurs.
– Tagc
Jul 2 at 13:20
I'm going to refrain from doing either of these for the time being because I'm still not clear what exact sequence of events can produce a bug, and I don't feel comfortable introducing code when I'm not clear on why exactly I'm doing it. For example, jeb's answer made it clear what kind of race condition would occur. But if you're right and the code still is unsafe, if it blows up again I'll have a good idea how to fix it.
– Tagc
Jul 2 at 13:49
While I upvoted your answer for the obvious dedication to make the question a good, SO-conformant one, your code still has one nasty bug which can surface in a couple of scenarios, being it a different optimization, a slightly different cycles/instruction CPU model or similar: your test for
countA >= countB
will yield (unintendedly, that is) true and therefore keep your function in the loop if you happen to read TCNT
fast enough to always hit the same value in both reads. This is a theoretical starvation scenario and a real source for unexplained glitches in execution time.– Vroomfondel
Jul 2 at 16:35
countA >= countB
TCNT
The atomic reads are not the main problem here.
It's the problem that the overflow-ISR and TCNT are highly related.
And you get problems when you read first TCNT and then the overflow counter.
Three sample situations:
TCNT=0x0000, Overflow=0 --- okay
TCNT=0xFFFF, Overflow=1 --- fails
TCNT=0x0001, Overflow=1 --- okay again
You got the same problems, when you change the order to: First read overflow, then TCNT.
You could solve it with reading twice the totalOverflow counter.
disable_ints();
uint16_t overflowsA=totalOverflows;
uint16_t cnt = TCNT;
uint16_t overflowsB=totalOverflows;
enable_ints();
uint32_t totalCnt = cnt;
if ( overflowsA != overflowsB )
{
if (cnt < 0x4000)
totalCnt += 0x10000;
}
totalCnt += (uint32_t)overflowsA << 16;
If the totalOverflowCounter changed while reading the TCNT, then it's necessary to check if the value in tcnt
is already greater 0 (but below ex. 0x4000) or if tcnt
is just before the overflow.
tcnt
tcnt
Ah yeah that actually does make sense. In a higher-level language I'd be locking around
TCNT
and _timerOverflows
together, but here I'm not sure how to short of manually constructing memory barriers in assembly, and like you say it's very possible that I read one variable and then the other just at the point that an overflow occurs.– Tagc
Jul 2 at 11:38
TCNT
_timerOverflows
@Tagc Even locking wouldn't help here, only temporarily stopping the counter would work, but that would produce inaccurate time results
– jeb
Jul 2 at 11:42
Believe me, if you have ever looked at HCS12 assembler generated from accessing unsigned longs, you would know that this is a major problem in the original code. In addition, failing to protect variables shared with an ISR is by far the most common embedded system newbie bug, and it always causes very subtle phenomenon.
– Lundin
Jul 2 at 11:45
@Lundin Once upon a time, I wrote code for the HCS12 ... And you are right, reading the overflow counter without disabling interrupts isn't a good idea. I added the disable_/enable_int() block
– jeb
Jul 2 at 11:54
@jeb It should only disable the specific timer channel, not the main interrupt mask. Which leads us back to the root of the problem: no can do since specific timer channels weren't used.
– Lundin
Jul 2 at 11:57
One technique that can be helpful is to maintain two or three values that, collectively, hold overlapping portions of a larger value.
If one knows that a value will be monotonically increasing, and one will never go more than 65,280 counts between calls to "update timer" function, one could use something like:
// Note: Assuming a platform where 16-bit loads and stores are atomic
uint16_t volatile timerHi, timerMed, timerLow;
void updateTimer(void) // Must be only thing that writes timers!
{
timerLow = HARDWARE_TIMER;
timerMed += (uint8_t)((timerLow >> 8) - timerMed);
timerHi += (uint8_t)((timerMed >> 8) - timerHi);
}
uint32_t readTimer(void)
{
uint16_t tempTimerHi = timerHi;
uint16_t tempTimerMed = timerMed;
uint16_t tempTimerLow = timerLow;
tempTimerMed += (uint8_t)((tempTimerLow >> 8) - tempTimerMed);
tempTimerHi += (uint8_t)((tempTimerMed >> 8) - tempTimerHi);
return ((uint32_t)tempTimerHi) << 16) | tempTimerLow;
}
Note that readTimer reads timerHi before it reads timerLow. It's possible that updateTimer might update timerLow or timerMed between the time readTimer reads
timerHi and the time it reads those other values, but if that occurs, it will
notice that the lower part of timerHi needs to be incremented to match the upper
part of the value that got updated later.
This approach can be cascaded to arbitrary length, and need not use a full 8 bits
of overlap. Using 8 bits of overlap, however, makes it possible to form a 32-bit
value by using the upper and lower values while simply ignoring the middle one.
If less overlap were used, all three values would need to take part in the
final computation.
The problem is that the writes to _timerOverflowsNoReset
isn't atomic and you don't protect them. This is a bug. Writing atomic from the ISR isn't very important, as the HCS12 blocks the background program during interrupt. But reading atomic in the background program is absolutely necessary.
_timerOverflowsNoReset
Also, have in mind that Codewarrior/HCS12 generates somewhat ineffective code for 32 bit arithmetic.
Here is how you can fix it:
volatile
Your code should be changed to something like this:
static volatile bool overflow;
void timovf_isr(void)
{
// Clear the interrupt.
TFLG2_TOF = 1;
// TEMP
overflow = true;
// ...
}
unsigned long MOTOR_GetCurrentTime(void)
{
bool of = overflow; // read this on a line of its own, ensure this is atomic!
uint16_t tcnt = TCNT; // read this on a line of its own
overflow = false; // ensure this is atomic too
if(of)
{
_timerOverflowsNoReset++;
}
/* calculations here */
return microseconds;
}
If you don't end up with atomic reads, you will have to implement semaphores, block the timer interrupt or write the reading code in inline assembler (my recommendation).
Overall I would say that your design relying on TOF is somewhat questionable. I think it would be better to set up a dedicated timer channel and let it count up a known time unit (10ms?). Any reason why you can't use one of the 8 timer channels for this?
I don't see how this code would work. Where do you reset
overflow
? Also, MOTOR_GetCurrentTime
shouldn't have side effects - your implementation will make it so that every time something queries the current time, the current time will change. "I think it would be better to set up a dedicated timer channel and let it count up a known time unit" I need to be able to keep track of time for several seconds at minimum for my application to work properly.– Tagc
Jul 2 at 11:11
overflow
MOTOR_GetCurrentTime
@Tagc Oops I missed adding the reset, fixed. But of course the current time will change each time you read the function, isn't that the whole point? TCNT is a hardware register. Regarding using the dedicated timer, keeping track of seconds is not a problem.
– Lundin
Jul 2 at 11:22
The idea of mutating state in
MOTOR_GetCurrentTime
seems problematic. It's like having a clock where the hands move every time you look at it. With your current implementation, wouldn't it lose time if several overflow interrupts occur in-between calls to MOTOR_GetCurrentTime
i.e. if timovf_isr
fires several times between calls to MOTOR_GetCurrentTime
?– Tagc
Jul 2 at 11:28
MOTOR_GetCurrentTime
MOTOR_GetCurrentTime
timovf_isr
MOTOR_GetCurrentTime
This fails, when TCNT is 0xFFFF you read
oveflow=false
, but uint16_t tcnt = TCNT
set tcnt=0
, you are missing the overflows at the edges– jeb
Jul 2 at 11:32
oveflow=false
uint16_t tcnt = TCNT
tcnt=0
@Tagc That's why I told you you need a soft real-time requirement of 22ms to service the routine.
– Lundin
Jul 2 at 11:35
It all boils down to the question of how often you do read the timer and how long the maximum interrupt sequence will be in your system (i.e. the maximum time the timer code can be stopped without making "substantial" progress).
Iff you test for time stamps more often than the cycle time of your hardware timer AND those tests have the guarantee that the end of one test is no further apart from the start of its predecessor than one interval (in your case 22ms), all is well. In the case your code is held up for so long that these preconditions don't hold, the following solution will not work - the question then however is whether the time information coming from such a system has any value at all.
The good thing is that you don't need an interrupt at all - any try to compensate for the inability of the system to satisfy two equally hard RT problems - updating your overflow timer and delivering the hardware time is either futile or ugly plus not meeting the basic system properties.
unsigned long MOTOR_GetCurrentTime(void)
{
static uint16_t last;
static uint16_t hi;
volatile uint16_t now = TCNT;
if (now < last)
{
hi++;
}
last = now;
return now + (hi * 65536UL);
}
BTW: I return ticks, not microseconds. Don't mix concerns.
PS: the caveat is that such a function is not reentrant and in a sense a true singleton.
With your implementation, what happens if multiple overflows occur between calls to
MOTOR_GetCurrentTime
? What happens if you call that function when TCNT = 30,000, then hits 65,535, resets to 0 and reaches 40,000 before the next call? "BTW: I return ticks, not microseconds. Don't mix concerns." This is a temporary function used for debugging - I have a very similar function that returns the time in ticks.– Tagc
Jul 2 at 12:41
MOTOR_GetCurrentTime
If multiple overflows occur, then the premise "test for time stamps more often than the cycle time of your hardware timer" is broken. In systems which really need time stamp of the delivered resolution, this is hardly ever a problem though. In the worst case one can test for the overflow flag of the hardware timer a issue a dummy read.
– Vroomfondel
Jul 2 at 12:46
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
I think you're over-complicating things. This approach should work: Atomically (i.e., within SEI/<your code>/CLI block, or if you don't know the current CCR[I] state: TPA/PSHA/SEI/<your code>/PULA/TAP) copy current counter and overflow, and clear both. Then, perform your calculations as you do but using the copies.
– tonypdmtr
Jul 2 at 16:43