tinyRTOS

A minimal preemptive RTOS for ARM Cortex-M, written from scratch to understand schedulers, context switching, TCBs and interrupts at the register and stack level. It runs on an STM32F103 Blue Pill with no HAL, my own linker script and startup code, and a context switch I wrote by hand in ARMv7-M assembly.

GitHub repo: tinyRTOS
Last update: 2026/07/18

Why I built it

tinyRTOS came out of my flight controller project. That control loop runs at 200 Hz and integrates a PID with a fixed DT = 0.005f, and for that constant to be true the loop has to run at exactly 5 ms. I get that guarantee by driving the loop from a hardware timer: the task blocks on ulTaskNotifyTake until the timer fires, then runs one iteration.

As I understood it, the reason I needed the hardware timer is that FreeRTOS on the ESP32 runs a 10 ms tick by default, and vTaskDelay only wakes a task on a tick boundary. A delay can’t be finer than the tick, so a 5 ms period isn’t expressible at all. That was a real problem, but not the real reason.

All I understood was that I had to route around it. I did not understand how it actually worked: what a tick actually is, what the scheduler does between my calls, how a context switch happens, and the real reason FreeRTOS’s vTaskDelay was inaccurate. So I built a small RTOS from scratch to find out.

Building it twice

The board was a couple of weeks out when I started, so I did as much as I could on Linux while waiting. That turned out to be the right way to build it regardless of the hardware delay.

On Linux I used ucontext, where swapcontext handles the actual register save and restore. The context switch stayed a black box I didn’t have to write yet, which let me get the whole scheduler logic correct first: TCBs, ready and blocked states, the tick, task_delay, round-robin picking. All of it debuggable with plain gdb, with no linker scripts or SWD.

Only after that was solid did I move to the STM32, where the one genuinely new thing was replacing swapcontext with a context switch I wrote myself. By then I understood the scheduler completely, so the hardware work was only the assembly. One hard thing at a time instead of all of them at once.

The repository keeps both. linux/ is the reference implementation with ucontext and a SIGALRM tick, stm32/ is the real thing.

The context switch

What resuming a task actually requires

Resuming a task means restoring its full CPU state, not just jumping to its address. Jumping to a function’s address restarts it from the top. Resuming means putting the CPU back exactly as it was: the program counter so execution continues on the right line, the stack pointer so its locals and call chain are intact, and every register it had live at the moment it was interrupted.

The task’s stack holds its variables and its return addresses, the whole record of who called whom. So if I preserve the stack and the registers, I’ve preserved everything.

Why the hardware saves half the registers and I save the other half

This follows the ARM calling convention. r0 to r3 and r12 are caller-saved scratch registers: any function may clobber them, so the caller preserves them if it needs them. r4 to r11 are callee-saved, meaning a function that uses them must restore them before returning.

An exception is an involuntary function call, so the hardware auto-stacks exactly the caller-saved set plus LR, PC and xPSR, the registers the interrupted code never got a chance to save itself. It leaves r4 to r11 alone because any handler that uses them saves them itself.

That works for ordinary interrupts, which return to the same code they interrupted. A context switch returns to a different task, whose r4 to r11 hold different values. So the switch has to save the outgoing task’s r4 to r11 and restore the incoming task’s by hand, which is why PendSV needs assembly rather than being a plain C function.

PendSV_Handler:
    // Save outgoing task
    MRS   r0, PSP
    STMDB r0!, {r4-r11}
    LDR   r1, =current_tcb
    LDR   r1, [r1]
    STR   r0, [r1]

    // Load incoming task
    LDR   r2, =next_tcb
    LDR   r2, [r2]
    LDR   r2, [r2]
    LDMIA r2!, {r4-r11}
    MSR   PSP, r2

    // current_tcb = next_tcb
    LDR   r0, =next_tcb
    LDR   r0, [r0]
    LDR   r1, =current_tcb
    STR   r0, [r1]

    BX    LR

The whole switch is those fifteen instructions. Read the outgoing task’s stack pointer out of PSP, push r4 to r11 onto it, store the resulting pointer into the TCB. Then walk to the incoming task’s saved stack pointer, pop r4 to r11 back, write the pointer into PSP, and update current_tcb. The final BX LR is an exception return, and the hardware unstacks the other eight registers on its way out.

sp is the first field of the TCB, at offset 0, so storing the stack pointer into the TCB is a single STR with no offset arithmetic. That was deliberate.

Starting a task that has never run

The context switch only knows how to resume a task: restore saved registers and continue. A new task has nothing saved.

The solution? You pretend it already ran, by forging its state manually.

So task_stack_init writes a fake exception frame onto its stack, the same sixteen words the hardware and the switch would have pushed if the task had been running and got interrupted.

higher address
&stack[256]  →  (top of stack)
                xPSR    = 0x01000000
                PC      = entry
                LR      = 0
                r12     = 0
                r3      = 0
                r2      = 0          eight words the hardware stacks
                r1      = 0
                r0      = 0
                r11     = 0
                r10     = 0
                r9      = 0
                r8      = 0          eight words PendSV stacks
                r7      = 0
                r6      = 0
                r5      = 0
                r4      = 0   ←  saved sp
lower address

Almost all of it is zeros, because a task that hasn’t run has no meaningful register values. Two entries matter. PC holds the task’s entry function, so the restore lands there. xPSR holds 0x01000000, which is bit 24, the Thumb bit.

Cortex-M executes only Thumb instructions, so that bit has to be set. If it isn’t, the CPU takes a fault on the task’s first instruction, and on bare metal that means jumping to the fault handler and hanging forever with no message. Just a frozen board.

Then the first switch into that task restores a state it never actually had, and execution begins at the top of its entry function.

Why handlers run on MSP and tasks on PSP

The Cortex-M has two stack pointers: MSP (main stack pointer, for handlers, kernel) and PSP (process stack pointer, for tasks). Three big reasons why this split exists:

Isolation: a task that overflows its stack corrupts its own memory rather than the kernel’s. Reliability: MSP is set up at reset and is always valid, so handlers always have a good stack no matter what any task did to its own. And what makes the design work at all: the hardware stacks the exception frame onto whichever stack was active when the exception fired. Tasks run on PSP, so an interrupted task’s frame lands on its own stack. That is what makes per-task save and restore possible, and it means switching tasks is just changing where PSP points. If everything ran on MSP, every task’s frame would pile onto one shared stack and there would be nothing to switch.

Why PendSV

A context switch has to happen at a clean boundary. PendSV is an exception you can request from software by setting a bit, and I set it to the lowest priority so it can never preempt another handler. It runs only once everything else has finished.

That guarantees a switch never happens nested inside another ISR, which would leave that handler half-finished on a stack I’m about to switch away from.

SysTick stays at high priority. It fires on time, decides who runs next, pends PendSV, and returns. Decide immediately at high priority, switch last at low priority.

void SysTick_Handler(void) {
        system_ticks++;
        scheduler_pick_next();
        if (next_tcb != current_tcb) {
                SCB_ICSR = PENDSVSET;
        }
}

What happened the first time I flashed it

The LED that was meant to blink stayed on, frozen.

That is one of the challenges of bare metal. There’s no printf, no stack trace, no exit code, so every mistake ends the same way: the CPU takes a fault, jumps to the fault vector, and my Default_Handler is while (1) { }. So the board just sits there. Several different bugs all produced one identical symptom.

What helped debugging was gdb over SWD. Attach with OpenOCD, continue, Ctrl-C, then p/x $pc and info symbol $pc, which tells you where it’s stuck. Then dump the scheduler’s own state with p system_ticks, p tcbs[0]->state, p current_tcb. Two commands to localise the problem, a few more to check what the bug actually was.

Some of the bugs are worth going over:

Stack alignment. task_stack_init started each task’s stack pointer at the top of its stack array. But sp is the first field of the TCB, so the array sits at offset 4, and depending on where each TCB landed in RAM some tasks got a 4-byte aligned initial SP instead of 8. ARM requires 8-byte alignment at exception entry. The fix is to mask off the low bits:

tcb->sp = (uint32_t *)(((uint32_t)&tcb->stack[256]) & ~0x7u);

Silent, layout dependent, and it would have faulted only on the tasks that happened to land wrong.

EXC_RETURN doesn’t work in thread mode. To launch the first task I tried the same trick PendSV uses: branch to the EXC_RETURN value 0xFFFFFFFD and let the hardware unstack the frame. That faults, as those values are only interpreted as an exception return when the CPU is in handler mode, and scheduler_start runs in thread mode, so it’s a branch to an invalid address and nothing more.

I replaced it with a manual launch. Read the entry PC out of the fake frame at offset 24, skip past the frame, set PSP, switch thread mode to PSP, and branch to the entry directly.

scheduler_start:
    LDR   r0, =current_tcb
    LDR   r0, [r0]
    LDR   r0, [r0]

    LDMIA r0!, {r4-r11}       // pop the callee-saved half
    LDR   r1, [r0, #24]       // entry PC, 7th of the 8 hardware-frame words
    ADD   r0, r0, #32         // skip the frame

    MSR   PSP, r0
    MOV   r0, #2              // CONTROL.SPSEL = 1, thread mode uses PSP
    MSR   CONTROL, r0
    ISB

    BX    r1

The Thumb bit rides in on bit 0 of the entry address, so the plain BX works without reconstructing xPSR.

The last one wasn’t a bug. After fixing everything the LED was still frozen, but under gdb a continue blinked it perfectly, so the code was correct. st-flash just wasn’t restarting the core after writing, so the board sat halted. Pressing the physical reset button made it blink, and st-flash --reset in the Makefile fixed it permanently. It was just not running, rather than not working.

One I didn’t find by debugging at all, only after reviewing the project: task_delay sets the task’s state to BLOCKED and then sets its wake time. SysTick can fire between those two lines, and if it does, the wake scan sees a task marked BLOCKED carrying a stale wake_time from its previous delay, marks it READY again, and the delay gets skipped silently.

This one isn’t a crash. It’s a rare, load-dependent wrong answer that no amount of watching an LED would ever reveal. The fix is a critical section:

void task_delay(uint32_t ticks) {
        irq_disable();
        current_tcb->wake_time = system_ticks + ticks;
        current_tcb->state = TASK_BLOCKED;
        scheduler_pick_next();
        SCB_ICSR = PENDSVSET;
        irq_enable();
}

cpsid i masks interrupts for the whole block-pick-pend sequence and cpsie i unmasks at the end. Pending PendSV while masked is fine because the request latches in hardware: the switch fires the moment interrupts come back on. It just can’t happen halfway through my bookkeeping.

What I understand now

Other than how an RTOS works in general, tasks and TCBs and the tick and what a context switch actually is, I started this to understand why vTaskDelay couldn’t pace a 5 ms control loop, and the real answer isn’t the one I initially assumed.

I assumed the issue was resolution. The ESP32’s FreeRTOS tick is 10ms by default, I wanted 5ms, so of course it didn’t fit. That’s true, and it’s the first limitation. A delay is built on the tick, so it can’t be finer than the tick and can only ever wake a task on a tick boundary. The tick interrupt is the only moment anything checks whether a delay has expired, and there’s no other point at which the question gets asked. My own task_delay works the same way and has the same property, just at 1ms instead of 10.

But a finer tick wouldn’t have fixed it, and that’s the part I only understood after building one. A delay is a relative minimum. A control loop needs an absolute period.

task_delay(5) means “don’t wake me for at least 5 ticks”, measured from the moment it is called. When those ticks elapse the task becomes ready, not running. It still has to be picked by the scheduler, which depends on what else is runnable. Then it does its work, which takes time, and only then calls task_delay(5) again. Each iteration is 5 ms plus the loop body plus whatever scheduling latency it hit, and because the next delay is measured from the end of that, the error keeps accumulating, it doesn’t just average out. The loop drifts, and how much it drifts depends on load.

A hardware timer solves it because it isn’t measuring from anywhere. It fires on a fixed grid every 5 ms, on its own clock, regardless of what the scheduler or my code is doing. The task blocks on ulTaskNotifyTake and the timer’s callback releases it. Nothing accumulates, because each edge is defined absolutely rather than relative to the last one. A tick-based delay measures a duration from wherever you happen to be. A hardware timer defines a grid you get pulled onto.

So building tinyRTOS didn’t give me a better delay, but gave me the reason the limitation exists, which is what turns the fix in my flight controller from something that empirically worked into something I actually understand and can explain.