Linux Kernel Subsystems

“In good software engineering, subsystems should not be arbitrary silos. They must interlock cleanly: the file system relies on the memory manager, the network stack passes buffers through the socket layer, and the scheduler orchestrates them all without deadlock.”

While the Linux kernel compiles into a single unified binary executing in Ring 0, its internal codebase is partitioned into five tightly integrated core subsystems.

Understanding how these subsystems collaborate under the hood reveals why Linux achieves near-zero latency, immense I/O bandwidth, and high-density virtualization.

+-------------------------------------------------------------------------------+
|                        System Call Interface (syscall)                        |
+-------------------+-------------------+-------------------+-------------------+
|   Process         |   Memory          |   Virtual File    |   Networking      |
|   Scheduler       |   Management      |   System (VFS)    |   Stack (NET)     |
|   (SCHED / EEVDF) |   (MM / SLUB)     |   (inodes/dcache) |   (sk_buff / XDP) |
+-------------------+-------------------+-------------------+-------------------+
|                        Inter-Process Communication (IPC)                      |
|                  Signals | Pipes | Sockets (AF_UNIX) | Futexes                |
+===============================================================================+
|                                PHYSICAL HARDWARE                              |
|                    CPU Cores | RAM | NVMe Storage | Network Cards             |
+-------------------------------------------------------------------------------+

1. Virtual File System (VFS): “Everything is a File”

The Virtual File System (VFS) is the kernel abstraction layer that allows client programs to interact with dozens of wildly different storage systems—ext4, XFS, Btrfs, NFS, FAT32, overlayfs, and pseudo-filesystems like /proc—using an identical, unified set of POSIX system calls (open(), read(), write(), close(), ioctl()).

The Four Primary VFS Data Structures (include/linux/fs.h):

1. struct super_block (The Filesystem Instance): Represents a mounted filesystem on a block device. Holds metadata about the filesystem: block size, max file size, dirty flags, and a pointer table to filesystem operations (super_operations).
2. struct inode (The Metadata Object): Represents a specific, unique file or directory object on disk. Contains file metadata: size, permissions (mode), owner (uid/gid), timestamps, access control lists, and pointers to the physical blocks on disk. Critically, an inode contains no filename. Multiple filenames can point to the same inode (hard links).
3. struct dentry (Directory Entry & The dcache): Represents a path component (e.g., in /usr/bin/python, usr, bin, and python each have a dentry). Dentries connect human-readable string pathnames to their underlying numerical inode. The kernel caches dentries in the dcache, using Read-Copy-Update (RCU) locks to achieve near-instantaneous path resolution without reading storage disks.
4. struct file (The Open File Descriptor): Represents an open file instance created in a process when calling open(). Tracks the current read/write byte offset (f_pos), access flags (O_RDONLY, O_NONBLOCK), and a pointer to the driver’s function table (file_operations).

The Linux Page Cache & Dirty Writeback:

When a program calls read() on a file, the kernel does not read directly from the NVMe or SSD. Instead:

• It checks if the requested 4KB data page is already cached in physical RAM within the Page Cache.
• If cached (Page Cache Hit), the read returns in nanoseconds without touching disk silicon.
• When writing data via write(), the kernel writes to RAM, marks the memory pages as “dirty”, and returns immediately. Background kernel flusher threads ([kworker] threads) flush dirty pages to physical persistent storage asynchronously.

2. Memory Management (MM)

The Linux Memory Management Subsystem is responsible for allocating, freeing, and mapping physical RAM across all running tasks while preserving strict process isolation.

1. The Buddy Allocator (Physical Page Allocation)

The Buddy Allocator manages physical memory in chunks of contiguous page frames ($2^0, 2^1, 2^2, \dots, 2^{10}$ pages).

• Standard page size on x86-64 is 4 KB (PAGE_SIZE = 4096).
• When the kernel needs an 8 KB block, it splits a 16 KB block into two 8 KB “buddies”.
• When memory is freed, adjacent buddy blocks automatically coalesce back into larger contiguous blocks, mitigating external fragmentation.

2. The SLUB Allocator (Kernel Object Caching)

While the Buddy Allocator works in 4KB page multiples, the kernel constantly allocates small objects (e.g., a 200-byte task_struct or an 80-byte inode). Allocating a whole 4KB page for an 80-byte struct would waste memory (internal fragmentation).

The SLUB allocator pre-allocates pools (caches) of fixed-size kernel objects:

• kmem_cache_create() sets up dedicated object pools.
• kmalloc() provides general-purpose byte-level kernel memory allocation.

3. Demand Paging & Page Faults

When an application calls malloc() or mmap() to request 100 Megabytes of memory, the kernel does not allocate physical RAM immediately. It merely updates the process’s Virtual Memory Area (VMA) tree.

Physical memory is allocated on-demand via Page Faults (#PF):

• Minor Page Fault: The virtual address is accessed for the first time. The MMU raises a page fault; the kernel intercepts it in do_page_fault(), grabs a physical page from the Buddy Allocator, populates the MMU page table entry, and resumes the process.
• Major Page Fault: The requested data is not present in RAM and must be read from disk or swapped in from NVMe storage. The process is put to sleep until the I/O transfer completes.
• Invalid Fault (SIGSEGV): The virtual address is outside any valid VMA or violates access permissions (e.g., writing to read-only code). The kernel sends SIGSEGV (Segmentation Fault) to terminate the process.

4. The Out-Of-Memory (OOM) Killer

When physical memory and swap space are completely exhausted, the kernel cannot fulfill allocations. To prevent a catastrophic total system freeze, the kernel invokes the OOM Killer:

• It evaluates all processes using an algorithm that scores their memory usage and runtime impact.
• It selects the process with the highest “badness score” and sends an uncatchable SIGKILL to reclaim its address space.
• Engineers can protect mission-critical daemons by tuning /proc/[pid]/oom_score_adj (setting it to -1000 immunizes a process from the OOM killer).

3. The Process Scheduler (SCHED)

The scheduler decides which runnable task executes on which CPU core, for how long, and when.

Scheduler Evolution:

1. Linux 2.4 $O(n)$ Scheduler: Scanned every runnable process in a linear loop on every tick. Stalled when process counts scaled past thousands.
2. Linux 2.6 $O(1)$ Scheduler: Authored by Ingo Molnar. Used priority-based active/expired bitmasks, guaranteeing constant-time selection regardless of process count. However, interactive heuristics were fragile.
3. CFS (Completely Fair Scheduler, 2007–2023): Modeled an ideal multi-tasking CPU on hardware. Tracked each process’s virtual runtime (vruntime) using a balanced Red-Black Tree. The task with the lowest vruntime was always scheduled next.
4. EEVDF (Earliest Eligible Virtual Deadline First, Linux 6.6+): Replaced CFS. Decouples task latency requirements from compute throughput. Tasks with short latency deadlines preempt throughput-heavy batch jobs without requiring hacky priority tweaks.

       EEVDF Runqueue (Sorted by virtual runtime & deadline)
             
                    [Task B: vruntime=120ms]
                           /         \
   [Task A: vruntime=90ms]             [Task C: vruntime=150ms]
             |
             v
   *Always picks lowest vruntime / earliest eligible deadline*

Scheduling Policies (include/uapi/linux/sched.h):

• SCHED_NORMAL (or SCHED_OTHER): Standard time-sharing policy used by virtually all general processes.
• SCHED_BATCH: For CPU-intensive background batch jobs; penalizes wake-up latency in favor of cache throughput.
• SCHED_FIFO (First-In, First-Out Real-Time): Executes until it voluntarily yields, blocks on I/O, or is preempted by a higher-priority real-time task.
• SCHED_RR (Round-Robin Real-Time): Like SCHED_FIFO, but tasks share equal time slices.
• SCHED_DEADLINE: Uses the Earliest Deadline First (EDF) algorithm for hard real-time guarantees in robotics and automotive systems.

4. The Networking Subsystem (NET)

In How Linux Works, Brian Ward dedicates multiple chapters to networking because Linux serves as the routing and packet-filtering backbone of modern cloud networks.

+-------------------------------------------------------------------------------+
|  USER SPACE: Application Sockets (HTTP, SSH, DNS, gRPC)                       |
+-------------------------------------------------------------------------------+
                                    |
                         [ POSIX Socket API: socket(), bind(), connect() ]
                                    v
+-------------------------------------------------------------------------------+
|  TRANSPORT LAYER (TCP / UDP): Connection state machines, congestion control   |
+-------------------------------------------------------------------------------+
                                    |
                         [ netfilter: INPUT / OUTPUT hooks ]
                                    v
+-------------------------------------------------------------------------------+
|  NETWORK LAYER (IPv4 / IPv6): Routing tables (ip route), TTL, Fragmentation   |
+-------------------------------------------------------------------------------+
                                    |
                         [ netfilter: PREROUTING / FORWARD / POSTROUTING ]
                                    v
+-------------------------------------------------------------------------------+
|  LINK LAYER & DEVICE DRIVERS: ARP resolution, sk_buff allocation, NAPI polling|
+-------------------------------------------------------------------------------+
                                    v
+-------------------------------------------------------------------------------+
|  PHYSICAL HARDWARE: NIC DMA Controller & Ring Buffers (XDP execution)         |
+-------------------------------------------------------------------------------+

1. struct sk_buff (Socket Buffer)

The socket buffer (sk_buff) is the most critical networking structure in the kernel:

• It represents a single network packet traversing the protocol stack.
• It avoids expensive memory copies. As a packet travels from the Ethernet driver → IP layer → TCP layer → Socket buffer, the kernel merely adjusts internal pointer offsets (head, data, tail, end) without copying packet payload bytes.

2. Network Interfaces & iproute2 Configuration

Linux abstracts hardware ports into network interfaces:

• lo: The Loopback interface (127.0.0.1), entirely software-driven in kernel memory.
• eth0 / enp3s0: Predictably named physical Ethernet controllers.
• veth pairs: Virtual ethernet pairs acting as bidirectional pipes between container network namespaces.

Superusers manage networking using modern iproute2 commands:

# Inspect link status and MAC addresses
ip link show

# Assign IP addresses
ip addr show

# Inspect kernel routing table and default gateway
ip route show

3. Netfilter Packet Filtering & NAT

Every packet entering or leaving the system traverses the kernel’s Netfilter framework across five well-defined hooks:

1. PREROUTING: Packets immediately after entering the NIC (used for Destination NAT / Port Forwarding).
2. INPUT: Packets destined for local processes on this machine.
3. FORWARD: Packets destined for another host (routing/firewall mode).
4. OUTPUT: Packets generated locally by user-space applications.
5. POSTROUTING: Packets about to leave the NIC (used for Source NAT / IP Masquerading).

Modern distributions govern Netfilter rules using nftables (replacing the legacy iptables table/chain system).

4. NAPI (New API) Interrupt Mitigation

Under high packet load (e.g., 10 million packets per second), generating a hardware CPU interrupt for every incoming packet would lead to receive livelock, where the CPU spends 100% of its time processing interrupts and zero time processing data.

Linux uses NAPI:

• When the first packet arrives, the network card triggers an interrupt.
• The kernel handles the interrupt, disables interrupts on the NIC, and transitions into a high-speed polling loop via ksoftirqd.
• Once the driver’s ring buffer is empty, interrupts are re-enabled.

5. eBPF & XDP (eXpress Data Path)

Modern cloud infrastructure frequently bypasses standard network stack overhead using XDP (eXpress Data Path):

• XDP executes sandboxed eBPF bytecode directly inside the network driver’s receive ring buffer, before an sk_buff is even allocated.
• Packets can be dropped (XDP_DROP), redirected (XDP_REDIRECT), or transmitted back (XDP_TX) in under 100 nanoseconds, defeating massive multi-terabit DDoS attacks at wire speed.

5. Inter-Process Communication (IPC)

Processes executing in isolated user spaces often need to coordinate, synchronize, and exchange data.

6. Hands-On Practical Labs

Lab 5.1: Analyzing Demand Paging & Page Faults

In Linux, calling malloc() does not allocate physical RAM; it merely reserves virtual address space. Physical frames are allocated on-demand upon the first write via a Minor Page Fault.

1. Page Fault Allocator: fault_test.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define ALLOC_SIZE (128 * 1024 * 1024) /* 128 MB */

int main(void) {
    printf("[*] PID: %d. Calling malloc(128MB)...\n", getpid());
    char *ptr = malloc(ALLOC_SIZE);
    if (!ptr) return 1;

    printf("[+] Virtual memory reserved. Sleeping 5s (check /proc/%d/status VmSize vs VmRSS)...\n", getpid());
    sleep(5);

    printf("[+] Writing data to pages. Hardware MMU page faults will trigger now...\n");
    for (size_t i = 0; i < ALLOC_SIZE; i += 4096) {
        ptr[i] = 'A'; /* Access each 4KB physical page */
    }

    printf("[+] All pages populated! Sleeping 5s (inspect VmRSS again)...\n");
    sleep(5);

    free(ptr);
    return 0;
}

2. Monitoring Page Faults in Terminal:

# Compile test
gcc -O2 fault_test.c -o fault_test

# Run in background
./fault_test &
PID=$!

# Monitor minor and major page faults every second
sar -B 1 10
# Alternatively:
# vmstat 1 10

Notice how fault/s spikes during the loop as the kernel handles 32,768 page faults (128MB / 4KB) and updates the page tables!

Lab 5.2: Zero-Permission File Descriptor Passing via AF_UNIX & SCM_RIGHTS

This lab demonstrates advanced Linux IPC: a privileged server opens /etc/shadow (or any protected file) and passes the active file descriptor directly to an unprivileged client over a Unix domain socket using sendmsg() and SCM_RIGHTS.

Complete Source: fd_pass.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <fcntl.h>
#include <sys/wait.h>

/* Helper to send open file descriptor over Unix socket */
void send_fd(int sock, int fd_to_send) {
    struct msghdr msg = {0};
    char buf[CMSG_SPACE(sizeof(int))];
    memset(buf, 0, sizeof(buf));

    struct iovec io = { .iov_base = "FD", .iov_len = 2 };
    msg.msg_iov = &io;
    msg.msg_iovlen = 1;
    msg.msg_control = buf;
    msg.msg_controllen = sizeof(buf);

    struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
    cmsg->cmsg_level = SOL_SOCKET;
    cmsg->cmsg_type = SCM_RIGHTS;
    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
    *((int *)CMSG_DATA(cmsg)) = fd_to_send;

    if (sendmsg(sock, &msg, 0) < 0) {
        perror("sendmsg failed");
        exit(1);
    }
}

/* Helper to receive open file descriptor from Unix socket */
int recv_fd(int sock) {
    struct msghdr msg = {0};
    char buf[CMSG_SPACE(sizeof(int))];
    char dummy[2];
    struct iovec io = { .iov_base = dummy, .iov_len = sizeof(dummy) };

    msg.msg_iov = &io;
    msg.msg_iovlen = 1;
    msg.msg_control = buf;
    msg.msg_controllen = sizeof(buf);

    if (recvmsg(sock, &msg, 0) < 0) {
        perror("recvmsg failed");
        return -1;
    }

    struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
    if (!cmsg || cmsg->cmsg_type != SCM_RIGHTS) {
        fprintf(stderr, "No SCM_RIGHTS payload received\n");
        return -1;
    }

    return *((int *)CMSG_DATA(cmsg));
}

int main(void) {
    int sv[2];
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0) {
        perror("socketpair");
        return 1;
    }

    pid_t pid = fork();
    if (pid == 0) {
        /* Child Process: Closes write socket, receives open fd from parent */
        close(sv[0]);
        printf("[Child] Waiting for transmitted file descriptor...\n");
        int received_fd = recv_fd(sv[1]);
        printf("[Child] Received FD: %d! Reading contents without open()...\n", received_fd);

        char read_buf[128];
        ssize_t n = read(received_fd, read_buf, sizeof(read_buf) - 1);
        if (n > 0) {
            read_buf[n] = '\0';
            printf("[Child] Read Content: %s\n", read_buf);
        }
        close(received_fd);
        close(sv[1]);
        exit(0);
    } else {
        /* Parent Process: Opens file and sends open fd to child */
        close(sv[1]);
        int fd = open("/etc/issue", O_RDONLY);
        if (fd < 0) fd = open("/etc/hostname", O_RDONLY);

        printf("[Parent] Opened file descriptor (%d). Transmitting via SCM_RIGHTS...\n", fd);
        send_fd(sv[0], fd);
        close(fd);
        close(sv[0]);

        wait(NULL);
    }
    return 0;
}

Compilation & Verification:

gcc -O2 fd_pass.c -o fd_pass
./fd_pass

Lab 5.3: Network Socket Lifecycle & Packet Inspection

In How Linux Works, superusers master diagnosing socket states and capturing traffic to verify kernel network routing.

# 1. Inspect listening TCP and UDP sockets with process PIDs
ss -tulpn

# 2. Inspect active TCP connections and socket timer states
ss -t -o state established

# 3. Trace kernel DNS resolution query flow
resolvectl status 2>/dev/null || cat /etc/resolv.conf

# 4. Live packet inspection: Capture ICMP ping packets on any interface
# Run this in one terminal:
sudo tcpdump -nn -i any icmp -c 4

# In another terminal:
# ping -c 2 127.0.0.1

7. Summary Checklist

• VFS Structs: Mastered super_block, inode, dentry, and file.
• Memory Management: Contrasted Buddy Allocator with SLUB, and understood demand paging.
• Process Scheduler: Analyzed CFS Red-Black trees and modern EEVDF lag compensation.
• Networking Stack: Explored sk_buff, NAPI polling loops, Netfilter hooks, and eBPF XDP line-rate filtering.
• Network Diagnostics: Inspected socket tables with ss and captured wire packets with tcpdump.
• Hands-on Labs: Tracked page faults with sar and implemented SCM_RIGHTS fd passing over AF_UNIX.