Kernel Drivers, Concurrency & LKML Development

9 min read 1616 words Edit on GitHub

“If you want to write kernel code, you need to understand that the kernel is an unforgiving environment. You have no libc, no dynamic stack growth, no memory protection from yourself, and if you corrupt a pointer or deadlock a spinlock, you bring down the entire operating system.”
— Linus Torvalds

Developing code inside Ring 0 requires a fundamental shift in programming mindset. In user space, a segmentation fault (SIGSEGV) terminates a single offending process. In kernel space:

  • There is no glibc or standard C library; only internal kernel primitives (kmalloc, printk, memmove).
  • The kernel stack is strictly limited (typically 8KB to 16KB per task), prohibiting large stack-allocated arrays.
  • Memory protection is absent: any rogue pointer write can corrupt the page tables, scheduling runqueues, or file buffers of the entire host.

This guide provides a comprehensive, production-grade walkthrough for developing Linux Loadable Kernel Modules (LKMs), writing character device drivers, implementing concurrency primitives, and formatting patches for upstream Linux Kernel Mailing List (LKML) submission.

1. Anatomy of a Character Device Driver

Device drivers bridge hardware peripherals or virtual abstractions to the Virtual File System. In Unix, character devices stream bytes sequentially (like serial ports, terminals, or /dev/urandom).

+--------------------------------------------------------------------------+
| USER SPACE: open("/dev/my_dev"), read(fd, buf, len), write(fd, buf, len)|
+--------------------------------------------------------------------------+
       | Traps via system call into Ring 0
       v
+--------------------------------------------------------------------------+
| VFS (Virtual File System)                                                |
|   Resolves path to struct inode -> struct cdev                           |
|   Extracts struct file_operations registered to Device Major:Minor       |
+--------------------------------------------------------------------------+
       |
       v Dispatches directly to your driver function pointers
+==========================================================================+
| YOUR KERNEL CHARACTER DRIVER                                             |
|                                                                          |
|   my_dev_open()    -> Allocates per-instance state                       |
|   my_dev_read()    -> Validates buffer, copy_to_user()                   |
|   my_dev_write()   -> copy_from_user(), acquires kernel mutex            |
|   my_dev_release() -> Cleans up resources                                |
+==========================================================================+

Major and Minor Numbers

Every device node in /dev is identified by a Major Number and a Minor Number (packed into a 32-bit dev_t type):

  • Major Number: Identifies the driver associated with the device (e.g., Major 1 for memory devices, Major 4 for TTYs).
  • Minor Number: Identifies the specific device instance managed by that driver.
/* Dynamic device number allocation */
dev_t dev_num;
alloc_chrdev_region(&dev_num, 0, 1, "my_char_device");
int major = MAJOR(dev_num);
int minor = MINOR(dev_num);

The file_operations Table

The kernel interfaces with your driver through function pointers defined in struct file_operations (defined in include/linux/fs.h):

static const struct file_operations my_fops = {
    .owner          = THIS_MODULE,
    .open           = my_dev_open,
    .release        = my_dev_release,
    .read           = my_dev_read,
    .write          = my_dev_write,
    .unlocked_ioctl = my_dev_ioctl,
};

2. Crossing User-Kernel Boundaries Safely

A classic vulnerability in kernel drivers is direct pointer dereferencing of user space addresses:

/* CRITICAL SECURITY BUG: NEVER DO THIS IN RING 0! */
char *user_ptr = (char *)arg;
*user_ptr = 'A'; /* May cause kernel panic, page fault, or Meltdown leak */

Why Direct Dereferencing Fails:

  1. The user space pointer could point to unmapped memory, triggering a page fault in an atomic context that deadlocks the kernel.
  2. The user space pointer could point into kernel address space (0xFFFF800000000000+), allowing an unprivileged user process to overwrite arbitrary kernel memory.

The Canonical Boundary Primitives:

#include <linux/uaccess.h>

/* Copy data from user space into kernel space buffer */
unsigned long copy_from_user(void *to, const void __user *from, unsigned long n);

/* Copy data from kernel space buffer to user space */
unsigned long copy_to_user(void __user *to, const void *from, unsigned long n);

Both functions return the number of uncopied bytes. A return value of 0 indicates complete success; any non-zero value indicates an invalid address (must return -EFAULT).

3. Kernel Synchronization & Concurrency

Linux is a preemptible Symmetric Multiprocessing (SMP) operating system. Multiple CPU cores can execute driver code simultaneously.

1. Spinlocks (spinlock_t)

  • Mechanism: The CPU loops in a tight polling loop (“spins”) until the lock becomes available.
  • Rules:
    • NEVER SLEEP while holding a spinlock! (Cannot call kmalloc(..., GFP_KERNEL), copy_to_user(), msleep(), or mutexes).
    • Used in interrupt handlers (top halves) and performance-critical low-latency paths.
    • Must use spin_lock_irqsave() if shared between interrupt context and process context to prevent single-core deadlocks.
spinlock_t my_lock;
spin_lock_init(&my_lock);

unsigned long flags;
spin_lock_irqsave(&my_lock, flags);
/* Critical section: cannot sleep! */
spin_unlock_irqrestore(&my_lock, flags);

2. Mutexes (struct mutex)

  • Mechanism: If the lock is held, the calling task is moved to TASK_UNINTERRUPTIBLE state, removed from the runqueue, and sleeps until awakened.
  • Rules:
    • Can only be used in process context (system calls, kernel threads).
    • Prohibited in interrupt context.
struct mutex my_mutex;
mutex_init(&my_mutex);

if (mutex_lock_interruptible(&my_mutex))
    return -ERESTARTSYS; /* Interrupted by signal */

/* Critical section: can sleep, copy to/from user, allocate memory */
mutex_unlock(&my_mutex);

3. Read-Copy-Update (RCU)

RCU is the cornerstone of high-performance Linux kernel scalability:

  • Readers: Acquire rcu_read_lock(), read shared data with zero atomic bus locks, and call rcu_read_unlock().
  • Writers: Allocate a fresh copy of the data, modify the copy, publish the pointer via rcu_assign_pointer(), and call synchronize_rcu() to wait for all pre-existing readers to finish before freeing the old memory.

4. Hands-On Practical Labs

Lab 10.1: Building a Thread-Safe Character Device Driver in C

In this lab, we develop, compile, and insert a complete character device driver (/dev/ringbuf_dev) that implements an in-kernel circular ring buffer with mutex synchronization.

1. Driver Source: ringbuf_dev.c

#include <linux/init.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/mutex.h>

#define DEVICE_NAME "ringbuf_dev"
#define CLASS_NAME  "ringbuf_class"
#define BUFFER_SIZE 256

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Linux Collective");
MODULE_DESCRIPTION("Thread-Safe Circular Buffer Character Device Driver");
MODULE_VERSION("1.0");

static dev_t major_minor;
static struct cdev char_cdev;
static struct class *char_class = NULL;
static struct device *char_device = NULL;

/* Device internal state */
struct ring_buffer_dev {
    char data[BUFFER_SIZE];
    size_t head;
    size_t tail;
    size_t count;
    struct mutex lock;
};

static struct ring_buffer_dev g_ring;

static int dev_open(struct inode *inodep, struct file *filep) {
    pr_info("ringbuf_dev: Device opened successfully\n");
    return 0;
}

static int dev_release(struct inode *inodep, struct file *filep) {
    pr_info("ringbuf_dev: Device closed\n");
    return 0;
}

static ssize_t dev_read(struct file *filep, char __user *buffer, size_t len, loff_t *offset) {
    size_t bytes_to_read;
    size_t i;
    char temp_buf[BUFFER_SIZE];

    if (mutex_lock_interruptible(&g_ring.lock))
        return -ERESTARTSYS;

    if (g_ring.count == 0) {
        mutex_unlock(&g_ring.lock);
        return 0; /* EOF */
    }

    bytes_to_read = min(len, g_ring.count);
    for (i = 0; i < bytes_to_read; i++) {
        temp_buf[i] = g_ring.data[g_ring.tail];
        g_ring.tail = (g_ring.tail + 1) % BUFFER_SIZE;
    }
    g_ring.count -= bytes_to_read;

    mutex_unlock(&g_ring.lock);

    /* Copy data safely to user space outside lock */
    if (copy_to_user(buffer, temp_buf, bytes_to_read))
        return -EFAULT;

    pr_info("ringbuf_dev: Read %zu bytes\n", bytes_to_read);
    return bytes_to_read;
}

static ssize_t dev_write(struct file *filep, const char __user *buffer, size_t len, loff_t *offset) {
    size_t bytes_to_write;
    size_t i;
    char temp_buf[BUFFER_SIZE];

    bytes_to_write = min(len, (size_t)BUFFER_SIZE);

    /* Copy data from user space first */
    if (copy_from_user(temp_buf, buffer, bytes_to_write))
        return -EFAULT;

    if (mutex_lock_interruptible(&g_ring.lock))
        return -ERESTARTSYS;

    for (i = 0; i < bytes_to_write; i++) {
        g_ring.data[g_ring.head] = temp_buf[i];
        g_ring.head = (g_ring.head + 1) % BUFFER_SIZE;
        if (g_ring.count < BUFFER_SIZE)
            g_ring.count++;
        else
            g_ring.tail = (g_ring.tail + 1) % BUFFER_SIZE; /* Overwrite oldest */
    }

    mutex_unlock(&g_ring.lock);

    pr_info("ringbuf_dev: Wrote %zu bytes\n", bytes_to_write);
    return bytes_to_write;
}

static const struct file_operations fops = {
    .owner   = THIS_MODULE,
    .open    = dev_open,
    .release = dev_release,
    .read    = dev_read,
    .write   = dev_write,
};

static int __init ringbuf_init(void) {
    int ret;
    pr_info("ringbuf_dev: Initializing character driver...\n");

    /* 1. Allocate dynamic major/minor number */
    ret = alloc_chrdev_region(&major_minor, 0, 1, DEVICE_NAME);
    if (ret < 0) {
        pr_err("ringbuf_dev: Failed to allocate major number\n");
        return ret;
    }

    /* 2. Initialize cdev structure */
    cdev_init(&char_cdev, &fops);
    char_cdev.owner = THIS_MODULE;

    ret = cdev_add(&char_cdev, major_minor, 1);
    if (ret < 0) {
        unregister_chrdev_region(major_minor, 1);
        pr_err("ringbuf_dev: Failed to add cdev\n");
        return ret;
    }

    /* 3. Create sysfs class and /dev node */
    char_class = class_create(CLASS_NAME);
    if (IS_ERR(char_class)) {
        cdev_del(&char_cdev);
        unregister_chrdev_region(major_minor, 1);
        return PTR_ERR(char_class);
    }

    char_device = device_create(char_class, NULL, major_minor, NULL, DEVICE_NAME);
    if (IS_ERR(char_device)) {
        class_destroy(char_class);
        cdev_del(&char_cdev);
        unregister_chrdev_region(major_minor, 1);
        return PTR_ERR(char_device);
    }

    /* Initialize internal mutex and ring state */
    mutex_init(&g_ring.lock);
    g_ring.head = 0;
    g_ring.tail = 0;
    g_ring.count = 0;

    pr_info("ringbuf_dev: Device registered successfully with Major: %d Minor: %d\n",
            MAJOR(major_minor), MINOR(major_minor));
    return 0;
}

static void __exit ringbuf_exit(void) {
    device_destroy(char_class, major_minor);
    class_destroy(char_class);
    cdev_del(&char_cdev);
    unregister_chrdev_region(major_minor, 1);
    mutex_destroy(&g_ring.lock);
    pr_info("ringbuf_dev: Device unregistered cleanly\n");
}

module_init(ringbuf_init);
module_exit(ringbuf_exit);

2. Kernel Module Makefile

obj-m += ringbuf_dev.o

KDIR ?= /lib/modules/$(shell uname -r)/build

all:
	make -C $(KDIR) M=$(PWD) modules

clean:
	make -C $(KDIR) M=$(PWD) clean

3. Compilation & Loading

# Compile against current kernel headers
make

# Insert module into Ring 0
sudo insmod ringbuf_dev.ko

# Verify device node creation in /dev
ls -l /dev/ringbuf_dev

# Inspect kernel log output
sudo dmesg | tail -n 10

Lab 10.2: Testing the Driver from User Space

Test writing and reading from your custom character device:

# Write data into the kernel buffer
echo "Linux Kernel Ring 0 Message" | sudo tee /dev/ringbuf_dev

# Read data out of the kernel buffer
sudo cat /dev/ringbuf_dev

Output:

Linux Kernel Ring 0 Message

Verify clean unloading:

sudo rmmod ringbuf_dev
sudo dmesg | tail -n 5

Lab 10.3: Formatting Upstream Patches for LKML & checkpatch.pl

The Linux kernel maintainer community strictly enforces code style standards. All patches must pass the checkpatch.pl validation script.

1. Commit Message Guidelines

A kernel commit message must follow this exact convention:

subsystem: brief summary in imperative mood (< 72 chars)

Explain the problem being solved. Include architectural context, 
why current code is flawed, and why this fix is correct.
Wrap lines at 72 characters.

Fixes: 1234567890ab ("original commit that introduced bug")
Signed-off-by: Your Name <your.email@domain.com>

2. Running checkpatch.pl

# Download or locate kernel checkpatch script
/usr/src/linux-headers-$(uname -r)/scripts/checkpatch.pl --strict --file ringbuf_dev.c

3. Generating the Upstream Patch

git format-patch -1 HEAD --subject-prefix="PATCH net-next"

5. Summary Checklist

  • Character Drivers: Mastered Major/Minor numbers, cdev, and file_operations.
  • Memory Safety: Protected kernel boundaries using copy_to_user and copy_from_user.
  • Concurrency: Applied Spinlocks in atomic context and Mutexes in process context.
  • RCU Synchronization: Understood lockless reads and grace period deferred reclamation.
  • Kernel Module Lab: Developed, compiled, and tested a complete character device driver in C.
  • LKML Patching: Practiced the upstream patch workflow and style validation with checkpatch.pl.