Bare-Metal RTOS for Raspberry Pi CM4

Delay

Non-blocking delay library

Overview

The delay library provides a pure NOP busy-wait delay function as well as a timed delay that uses the scheduler tick count. Neither yields the CPU to another task.

API

Status

The single StatusCode enum that almost every function in the kernel returns.

Mutex

A priority-inheritance mutex implementation.

How it works

Each Mutex carries its own Spinlock since a waiter can live on a different core than the owner. mutex_lock() is managed by the scheduler systick. If inheritance is enabled and the caller is higher priority than the current owner, the owner's priority is boosted for as long as it holds the mutex.

struct Mutex {
  MutexState         state;               // locked or unlocked
  TaskControlBlock  *mutex_owner;         // task currently holding it
  List               mutex_blocked_list;  // tasks waiting
  uint8_t            inheritance_enabled; // priority inheritance on/off
  TaskPriorityLevel  inherited_priority;  // highest priority among current waiters
  Spinlock           lock;                // cross-core lock on this struct
#ifdef RTOS_TELEMETRY
  uint16_t           sync_id;             // telemetry builds only
#endif
};

Mutex control flow

What mutex_lock() and mutex_unlock() do on every call:

Priority inheritance example

A worked example of the boost across two cores, Task A holding the mutex on core 0 while a higher-priority Task B blocks on core 1:

Finding the next owner

mutex_unlock() finds the next owner and the second-highest waiting priority in a single pass over the blocked list:

while (iter != NULL) {
  if (iter->owner->priority > best->owner->priority) {
    if (best->owner->priority > second_best) second_best = best->owner->priority;
    best = iter;
  } else {
    if (iter->owner->priority > second_best) second_best = iter->owner->priority;
  }
  iter = iter->next;
}

API

Semaphore

A counting semaphore for signaling between tasks and interrupt handlers.

Overview

semaphore_take() blocks by adding the task to a FIFO list and blocking until something wakes it - either semaphore_give() or a timeout from the scheduler tick. semaphore_give() hands its count directly to the next waiting task instead of incrementing the counter and letting it race to claim it. give() also locks the waiting task's own core, not the caller's core, so it's safe to call from a task or ISR on one core to wake a task sitting on another core's ready list.

struct Semaphore {
  uint32_t max_count;                // ceiling
  uint32_t count;                    // current value
  List     semaphore_blocked_list;   // FIFO wait list
  Spinlock lock;                     // cross-core lock on this struct
#ifdef RTOS_TELEMETRY
  uint16_t sync_id;                  // telemetry builds only
#endif
};

Semaphore take & give execution

Semaphore take with timeout

semaphore_take() can also be called with a timeout parameter so that the caller is guaranteed to return after a certain amount of time. If the timeout expires without the semaphore being acquired, then semaphore_give() will return E_TIMED_OUT

API

Queue

A fixed-capacity message queue built on top of two semaphores.

Overview

queue_init() allocates its backing buffer with one heap_malloc() call, then sets up two semaphores: space_available starts at full capacity, data_available starts at 0 — the standard producer/consumer pattern. queue_send()/queue_recv() copy an item in or out byte-by-byte (no libc, no memcpy in this freestanding build) and have no wait-list logic of their own; both blocking paths go through semaphore.

struct Queue {
  uint8_t  *buf;                   // backing buffer, capacity * item_size bytes
  uint32_t  head;                  // write index
  uint32_t  tail;                  // read index
  uint32_t  capacity;              // max number of items in the queue
  uint32_t  item_size;             // bytes per item

  Semaphore space_available;       // gates queue_send
  Semaphore data_available;        // gates queue_recv

  Spinlock  lock;                  // guards the byte-copy step
#ifdef RTOS_TELEMETRY
  uint16_t  sync_id;                // telemetry builds only
#endif
};

Ring buffer operation:

Queue send/receive operations

Sized for any item type

queue_init() takes item_size as a parameter and allocates capacity * item_size bytes for the backing buffer. queue_send()/queue_recv() don't know or care what type is being queued — src and dst are void *, and both copy exactly item_size bytes to or from the buffer at head/tail's offset. The same Queue works for single bytes or large structs; only the numbers passed to queue_init() change.

API

Software Timer

Tick-driven timers built on top of the scheduler. All software timers live in a software timer handler task. Software timers can be configured to have one-shot or repeating callbacks.

Overview

Each SoftwareTimer sits in one of two lists: a blocked list sorted by expiry_tick, and an active list. software_timer_tick() runs in on the scheduler systick and moves expired timers onto the active list and signals a semaphore. The callback itself runs later, off the tick, on a dedicated sw_timer service task. Periodic timers re-arm relative to their scheduled expiry (expiry_tick += period), not to when they actually ran, to prevent drift.

struct SoftwareTimer {
  uint64_t       expiry_tick;   // scheduler tick this timer next fires at
  uint64_t       period;        // ticks between fires (periodic) or until fire (one-shot)

  TimerMode      timer_mode;    // one-shot or periodic
  TimerCallback  callback;      // runs when it fires

  TimerListId    list_id;       // which list this node is currently in
  SoftwareTimer *next;          // intrusive singly-linked list link
};

Software timer lifecycle

API

Spinlock

A cross-core lock using Lamport's Bakery algorithm.

Overview

Each core draws a ticket one higher than the max ticket any other core currently holds, then waits until every other core whose (ticket, core_id) pair sorts before its own has passed through.

Spinlock usage

Multicore example

API

UART

UART driver with RX calls that automatically switch between blocking and task-based transmission.

Overview

uart_tx() operations all check uart_task_started before every write. Until a channel's TX task has been started, they fall through to raw blocking implementation, which happens during early boot, before the scheduler or task exists. Once uart_channel_task_start() has been called for a channel, the same calls instead enqueue into a ring buffer and wake a dedicated uart_tx_task with a semaphore. The UART buffer + task mechanism is guarded by a spinlock and can be accessed from any core

UART buffer + task mechanism

The UART buffer and task mechanism keeps uart calls lightweight which is important for an RTOS. Additionally, it prevents jumbled/stomped UART messages when multiple cores & tasks try to write through the same UART peripheral.

kmain()
  uart_channel_init(...)
  uart_print("booting...")       -> raw PIO, straight out
  scheduler_init(...)
  uart_channel_task_start(...)   -> creates uart_tx_task
  scheduler_start()
  ...
  uart_print("running")          -> same call, now enqueues
                                     + wakes uart_tx_task

Channel table

Six BCM2711 UART instances exist; one is unsupported and the rest are fixed hardware facts, not caller-configurable:

Configuring a channel

Each channel gets its own independent UartConfig, so pins, baud rate, and mode can differ freely from channel to channel.

typedef struct {
  uint8_t      tx_pin;             // GPIO pin for TX
  uint8_t      rx_pin;             // GPIO pin for RX, or UART_PIN_NONE for TX-only
  GPIOFunc     alt_func;           // ALT function tx_pin/rx_pin get muxed to
  UartBaudrate baudrate;           // UART_BAUDRATE_115200 or UART_BAUDRATE_921600
  UartMode     mode;               // UART_MODE_BLOCKING or UART_MODE_BUFFERED_TASK
  bool         is_dma_enabled;     // only meaningful when mode == UART_MODE_BUFFERED_TASK
  uint8_t      dma_channel;        // which DMA channel to use if is_dma_enabled
  uint16_t     task_stack_words;   // TX drain task's stack size
  uint8_t      task_priority;      // TX drain task's priority
} UartConfig;

Configuring the print channel for buffered, DMA-paced TX:

static UartConfig print_uart_cfg = {
  .tx_pin = 14, .rx_pin = 15,
  .alt_func = GPIO_FUNC_ALT0,
  .baudrate = UART_BAUDRATE_115200,
  .mode = UART_MODE_BUFFERED_TASK,
  .is_dma_enabled = true,
  .dma_channel = UART_DEFAULT_DMA_CHANNEL,
  .task_stack_words = 256,
  .task_priority = TASK_PRIORITY_2,
};   // static — uart_channel_init() stores this pointer, not a copy

uart_channel_init(UART_CHANNEL_PRINT, &print_uart_cfg);
uart_channel_task_start(UART_CHANNEL_PRINT);

API — lifecycle

API — TX

API — RX

API — DMA & diagnostics

GPIO

Direct register-level control of the BCM2711 GPIO controller — function select, pull, and pin I/O.

API

CRC

CRC32 computed with the ARMv8 hardware crc32b/h/w instructions.

API

DMA

BCM2711 legacy DMA controller used to allow other drivers to move data asynchronously.

Users declare a DmaControlBlock struct and register an ISR to unblock when the transaction is complete. The heaviest consumer of the DMA controller is the UART driver.

typedef struct {
  uint32_t ti;          // transfer information (DMA_TI_* flags, DREQ pacing)
  uint32_t source_ad;   // source bus address
  uint32_t dest_ad;     // destination bus address
  uint32_t txfr_len;    // transfer length in bytes
  uint32_t stride;      // 2D stride (0 in linear mode)
  uint32_t nextconbk;   // bus address of next CB, or 0 to stop
  uint32_t reserved[2];
} __attribute__((aligned(32))) DmaControlBlock;   // 8 words, DMA-visible RAM

How uart_tx_task uses it

When a UART controller optionally enables DMA usage, the uart_tx task will instead copy the desired uart buffer address to a new DmaControlBlock, start the DMA operation, and wait on a semaphore. When the hardware finishes the DMA_IRQ fires and gives the semaphore back. Since the uart buffer is a ring buffer, in some cases where the buffer contents wrap around the physical array boundary, the DMA transaction is automatically split in two.

API

eMMC

The eMMC flash memory controller driver is used by the bootloader and app to read/write persistent storage (boot flags, bootloader & app binaries).

API

GIC & Interrupts

GIC-400 interrupt routing, and a dispatch table every registered handler goes through.

Initializing a new interrupt and interrupt execution flow:

GIC initialization: gic_distributor_init() runs once globally. Every core, including core 0, must separately call gic_percore_init() to bring up its own banked CPU interface.

IRQ registration: irq_register(intid, handler) populates a flat 256-entry table (g_handlers[intid]) that the assembly _irq_handler calls into via irq_dispatch(intid).

ISR routing: When an interrupt fires, the PC jumps to the _irq_handler located in assembly startup code. It executes a context save and jumps to the appropriate ISR.

API

Generic Timer

The physical timer (CNTP) that drives the 1 kHz scheduler tick.

Overview

gentimer_init() programs CNTP_CVAL for the first tick and enables the timer; read_cntpct() reads the free-running 64-bit physical counter.

The generic timer interrupt is treated as a special case by _irq_handler in assembly, directly calling (cntx_switch$) instead of routing it through the generic C dispatch table like every other interrupt.

API

I2C

Generic I2C driver with access to all six usable I2C channels.

i2c_channel_write()/read() run a simple poll FIFO → poll DONE sequence.

typedef struct {
  uint8_t     sda_pin;
  uint8_t     scl_pin;
  GPIOFunc    alt_func;   // ALT function sda_pin/scl_pin get muxed to
  I2cBaudrate baudrate;   // I2C_BAUDRATE_STANDARD_100K or I2C_BAUDRATE_FAST_400K
} I2cConfig;

API

I2S

A full-duplex PCM/I2S driver

i2s_transfer() services both the TX and RX FIFOs in one interleaved polling loop rather than separate write/read calls, since RXON keeps sampling regardless of whether software is draining it.

API

JTAG

One function that enables hardware debugging.

API

Mailbox

mbox_property_call() writes a bus address into MBOX_WRITE and polls MBOX_READ until the same address echoes back. The request buffer and the response occupy the same memory, in place. mbox_get_clock_rate(), mbox_get_board_serial(), and mbox_get_arm_memory() are one-tag convenience wrappers around it.

API

PWM (PCA9685)

A driver for the PCA9685 16-channel I2C PWM controller, used for servos and LEDs.

pwm_pca9685_init() wakes the chip from sleep, computes PRE_SCALE from the target update rate via the datasheet formula, and restarts it with auto-increment enabled. pwm_pca9685_set_channel() sets a channel's on/off tick positions from delay/duty cycle fractions.

API

Reset

Two different ways to reboot - hard reset and soft reset into bootloader.

enter_bootloader() zeros the entire .stacks region, including the IRQ-mode stack, then jumps to the bootloader. This is used during DFU.

API

Watchdog

A watchdog driver that monitors the health of the RTOS, manages boot flags and supports multicore mode.

Overview

watchdog_init() arms the BCM2711 PM watchdog (timeout clamped to 1–15s) from a caller-owned WatchdogConfig. The watchdog is kicked through a dedicated watchdog task.

If configured, the watchdog features a confirm-slot timer that clears the current app's "on trial" flag once it's run long enough to be trusted. wdt_meta (persisted to eMMC, CRC-checked) tracks reset count, active slot, and trial state across reboots.

Watchdog config struct:

typedef struct {
  uint32_t timeout_s;                         // clamped to [1, 15]s by watchdog_init()
  WatchdogResetPolicy policy;                 // what the bootloader does past tolerance
  int32_t  tolerance;                         // number of watchdog failures until the bootloder steps in
  int64_t  confirm_delay_ms;                  // WATCHDOG_CONFIRM_MANUAL = confirm yourself
  bool     multicore_mode;                    // aggregate kicks from every companion core
  CompanionCoreContext *companion_core_ctx;   // required when multicore_mode is true
} WatchdogConfig;

Watchdog boot flags:

typedef struct {
  uint32_t magic;               // WDT_META_MAGIC; mismatch -> struct discarded
  uint32_t wdt_reset_count;     // consecutive watchdog resets since last confirm
  int32_t  wdt_reset_tolerance; // copy of WatchdogConfig.tolerance, persisted
  uint32_t wdt_reset_policy;    // copy of WatchdogConfig.policy, persisted
  uint32_t wdt_reset_reason;    // not yet written by the bootloader
  uint32_t active_app_slot;     // APP_SLOT_A / APP_SLOT_B (see eMMC page)
  uint32_t app_slot_trial;      // 1 = active slot unconfirmed (A/B trial)
  uint32_t trial_boot_count;    // bootloader re-entries while on trial
  uint32_t crc;                 // CRC32 over every field above; MUST be last
} WdtMeta;   // lives in eMMC EMMC_SECTOR_METADATA; wdt_meta is the in-RAM shadow

Multicore kick aggregation

When multicore_mode is on, watchdog_core_kick() doesn't touch the hardware watchdog directly, it's guarded. This guarantees that one failing core triggers a watchdog reset. The watchdog keeps track of which cores are initialized via the compantion core context struct.

Confirm-slot two-stage design

The watchdog supports the bootloader's app-slotting feature. A one shot timer callback signals a semaphore, and a separate dedicated task performs the eMMC write that signals to the bootloader that the app is healthy..

API

CPRMAN

Peripheral clock source for I2C, UART, and I2S.

API

Scheduler

A 6-level priority, round-robin scheduler, with one instance per core.

Overview

Each core owns its own scheduler_init() call, its own ready/blocked lists, its own idle task, and its own Spinlock scheduler lock. The current implemention is based on a BMP (bound multiprocess) model, so each task is bound to a core. core_id never changes once created.

SchedulerCore struct, one owned per core:

typedef struct {
  List               ready_list[NUM_TASK_PRIORITIES]; // one per priority level
  List               blocked_task_list;               // sorted by wakeup_time
  volatile uint32_t *s_clk_freq;                       // this core's timer frequency
  volatile uint64_t *s_tick_count;                     // this core's tick counter
  uint32_t           hz;                               // tick rate (1000)
  Spinlock           lock;                             // this core's scheduler lock
  TaskControlBlock   idle_tcb;                         // this core's idle task
  StackType_t        idle_stack[IDLE_STACK_DEPTH];     // idle task's stack (64 words)
} SchedulerCore;

static SchedulerCore g_cores[COMPANION_CORE_MAX_CORES];  // one per core, indexed by core_id

Task States

The scheduler is responsible for transitioning tasks between READY, RUNNING, and BLOCKED. Calling mutex_lock(), semaphore_take(), or task_delay_ms() pulls the task off its priority's ready list, drops it onto the blocked_task_list sorted by wakeup_time.

Scheduler Systick execution

scheduler_switch_context() runs on every 1 kHz tick and executes the round robin algorithm. Starting from the highest priority level and working down, the scheduler picks the first non-empty ready list and runs whichever task list->index currently points at. Several tasks sharing a priority rotate.

API

Tasks

Units of execution defined by the system.

Overview

task_create() allocates a TaskControlBlock from a fixed per-core pool (16 slots per core, MAX_NUM_TASKS) and a stack from the kernel heap, fills the stack with a watermark byte, and adds the task to its core's ready list.

TaskControlBlock struct

Each task is identified by a TCB struct, which keeps track of priority, stack location, task state, and tracks if the task is part of a ready or blocked list.

struct TaskControlBlock {
  volatile StackType_t *current_sp;  // live SP, saved/restored on every switch
  StackType_t          *stack_high;  // top of allocated stack (initial SP)
  StackType_t          *stack_base;  // bottom of stack; watermark check
  uint16_t              stack_depth;

  uint16_t              task_id;
  uint32_t              core_id;     // which core's scheduler owns this task

  volatile TaskState    current_state;
  TaskPriorityLevel      priority;
  uint64_t               wakeup_time;

  ListItem               state_list_item;  // ready/blocked list node
  ListItem               event_list_item;  // mutex/semaphore wait node

  volatile TaskWakeupReason wakeup_reason;

  TaskPriorityLevel      base_priority;    // pre-boost priority
  uint32_t               mutexes_held;     // inheritance-enabled mutexes held
};

Task states

Tasks transition between three main states:

Task Stack:

task_init_stack() builds a task's very first stack frame, which is essential to context switching.

Creating a task

A task function takes one void * parameter and, typically never return. task_exit_trap() exists specifically to catch one that does.

static TaskControlBlock *blink_tcb = NULL;

static void blink_task(void *params)
{
  (void)params;
  while (1) {
    gpio_on(LED_PIN);
    task_delay_ms(500);
    gpio_off(LED_PIN);
    task_delay_ms(500);
  }
}

// after scheduler_init(), before scheduler_start():
StatusCode ret = task_create(blink_task, 256, TASK_PRIORITY_2, NULL, "blink", &blink_tcb);
if (ret != E_OK) {
  uart_printf("task_create failed: %d\r\n", ret);
}

API

Heap

A 256 KB bump allocator to manage dynamic memory in the kernel. Access is guarded by a spinlock.

Overview

static uint8_t   heap[HEAP_SIZE_BYTES];   // 256 KB, HEAP_SIZE_BYTES = 262144
static uint32_t  heap_offset = 0;
static Spinlock  heap_lock;               // guards heap_offset across cores

void *heap_malloc(uint32_t size)
{
  size = (size + 3) & ~0b11;              // round up to 4-byte alignment

  if (heap_offset + size > HEAP_SIZE_BYTES) {
    return NULL;
  }
  void *block_start = &heap[heap_offset];
  heap_offset += size;
  return block_start;
}

API

Companion Core

Releases cores 1–3 from their boot-time parking loop into independent RTOS instances.

Overview

companion_core_start(core_id, entry) hands a secondary core a bare function pointer and wakes it with a SEV.

CompanionCoreContext struct

The CompanionCoreContext struct is mainly used to manage the multicore watchdog feature

typedef struct {
  volatile uint32_t expected_mask;          // which cores should report in
  volatile uint8_t  core_kicked[COMPANION_CORE_MAX_CORES]; // per-core kick flags
} CompanionCoreContext;

Companion core startup sequence:

What a released core actually does with that entry point is run its own complete, independent scheduler_init()/task_create()/scheduler_start() sequence: a second full instance of everything in the Base of RTOS / Kernel tier above.

A companion core's entry function

static volatile uint32_t core1_clk_freq;
static volatile uint64_t core1_tick_count = 0;

static void led_task(void *params)
{
  (void)params;
  while (1) {
    gpio_on(LED_PIN);
    task_delay_ms(500);
    gpio_off(LED_PIN);
    task_delay_ms(500);
  }
}

static void core1_kmain(void)
{
  gpio_set_function(LED_PIN, GPIO_FUNC_OUTPUT);

  gic_percore_init();                              // this core's own CPU interface
  gentimer_init(&core1_clk_freq, hz);               // this core's own tick source
  scheduler_init(1U, &core1_clk_freq, hz, &core1_tick_count);

  TaskControlBlock *tcb;
  task_create(led_task, 2048, TASK_PRIORITY_1, NULL, "led", &tcb);

  __asm__ volatile ("cpsie i" ::: "memory");        // this core's own IRQs
  scheduler_start();                                // never returns
}

Core 0 releasing it

void kmain(void)
{
  ...
  gic_distributor_init();   // global — must happen before any core is released
  gic_percore_init();       // core 0's own PPI30 + CPU-interface enable
  gentimer_init(&clk_freq, hz);

  if (companion_core_start(1U, core1_kmain) == E_OK) {
    uart_print("core 0: released core 1\r\n");
  }

  __asm__ volatile ("cpsie i" ::: "memory");
  scheduler_start();
}

API