Storage Architecture, Filesystems & io_uring

12 min read 2236 words Edit on GitHub

“The fastest I/O is the I/O you don’t do. The second fastest is the I/O that doesn’t trap into the kernel with a system call on every single buffer.”
— Linus Torvalds

Modern NVMe solid-state storage can execute millions of Input/Output Operations Per Second (IOPS) with microsecond latencies. However, traditional POSIX system call interfaces—such as read(), write(), and even multiplexers like epoll—were architected in an era of rotational magnetic disks where rotational latency (milliseconds) dwarfed the CPU overhead of system calls and context switches.

On modern hardware, the overhead of trapping from Ring 3 to Ring 0, allocating kernel data structures, and triggering interrupts is often slower than the physical storage media itself.

This guide explores the Linux storage stack from the Virtual File System (VFS) down to the Block I/O Layer, investigates the Page Cache, and walks through the io_uring asynchronous I/O interface created by Jens Axboe.

1. The Linux Storage Pipeline

When a user space application issues a write request, data journeys through multiple architectural layers before touching physical non-volatile silicon:

+--------------------------------------------------------------------------+
| USER SPACE: Application (Databases, Web Servers, Compilers)              |
+--------------------------------------------------------------------------+
       | read() / write() / pwritev2()             | io_uring SQ/CQ Rings
       v                                           v (Zero syscalls via SQPOLL)
+--------------------------------------------------------------------------+
| KERNEL SPACE: Virtual File System (VFS)                                  |
|   struct file -> struct inode -> struct dentry                           |
+--------------------------------------------------------------------------+
       |                                           |
       v (Buffered I/O)                            v (Direct I/O: O_DIRECT)
+------------------------------------+             |
| Page Cache (struct address_space)  |             |
|   Dirty Pages / [kworker] threads  |             |
+------------------------------------+             |
       |                                           |
       +---------------------+---------------------+
                             v
+--------------------------------------------------------------------------+
| Concrete Filesystem (Ext4, XFS, Btrfs, ZFS)                              |
|   Translates logical file offsets into physical block sector addresses   |
+--------------------------------------------------------------------------+
       v
+--------------------------------------------------------------------------+
| Block I/O Layer & Schedulers                                             |
|   struct bio allocation, Request Merging (mq-deadline, bfq, none)        |
+--------------------------------------------------------------------------+
       v
+--------------------------------------------------------------------------+
| Device Driver Layer (nvme.ko, ahci.ko, virtio_blk.ko)                    |
|   Hardware Submission Queues & Completion Queues via DMA                 |
+--------------------------------------------------------------------------+
       v
+--------------------------------------------------------------------------+
| PHYSICAL HARDWARE: PCIe NVMe SSD / SATA Controller                       |
+--------------------------------------------------------------------------+

2. Virtual File System & Page Cache Mechanics

1. Buffered I/O vs. Direct I/O (O_DIRECT)

By default, all Linux file operations are Buffered I/O:

  • Reads: If requested data resides in the kernel’s Page Cache, the CPU copies data from kernel memory to the user space buffer via copy_to_user(). No disk access occurs. If absent, a Major Page Fault triggers synchronous disk retrieval.
  • Writes: When an application writes, data is placed directly into the Page Cache and marked as Dirty. The write() system call returns immediately to user space. Background kernel flusher threads ([kworker]) asynchronously serialize dirty pages to disk.
/* Direct I/O flags in <fcntl.h> */
int fd = open("/var/lib/data.db", O_RDWR | O_DIRECT | O_SYNC);

When O_DIRECT is passed:

  1. The kernel bypasses the Page Cache entirely.
  2. The user space buffer must be memory-aligned (typically to 512-byte or 4096-byte boundaries).
  3. The DMA controller transfers data directly between the user space buffer and the storage controller, preventing double-buffering in RAM for databases like PostgreSQL and RocksDB.

2. Tuning Page Cache Writeback Ratios

Under sustained heavy write workloads, unwritten dirty pages accumulate in RAM. If RAM fills with dirty pages, Linux will stall user processes until disk flushes finish.

Key kernel sysctl parameters under /proc/sys/vm/:

  • vm.dirty_background_ratio (Default: ~10%): Percentage of system memory containing dirty pages at which background [kworker] threads wake up to write data to disk.
  • vm.dirty_ratio (Default: ~20%): Hard ceiling. If dirty pages exceed this percentage, all producing user space threads are blocked and forced to synchronously flush data to disk.

3. Filesystem Anatomy: Inodes, Dentries & Journaling

In How Linux Works, Brian Ward emphasizes that understanding the difference between an inode, a directory, and a data block is the single most critical concept in storage engineering.

+-------------------------------------------------------------------------+
|    DIRECTORY FILE: "/var/log" (List of filename -> inode_number pairs)  |
|      "syslog"   -> Inode #1048577                                       |
|      "messages" -> Inode #1048577  (Hard link! Same inode)              |
|      "nginx"    -> Inode #2097152                                       |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|    INODE #1048577: File Metadata                                        |
|      - File type (regular file, directory, socket)                      |
|      - Permissions (0644), Owner (root:adm)                             |
|      - Timestamps (atime, mtime, ctime)                                 |
|      - Size (24,576 bytes)                                              |
|      - Link Count (i_nlink = 2)                                         |
|      - Extent Tree -> Pointers to Physical Disk Blocks [1024, 1025...]  |
|      *(NOTICE: THE INODE DOES NOT STORE THE FILENAME!)*                 |
+-------------------------------------------------------------------------+

Inodes vs. Directories:

  • Inode: The fundamental metadata record for a file. It contains ownership, permissions, size, and pointers to storage blocks, but never the file name.
  • Directory: A directory is simply a special file containing an array of directory entries (dentry). Each entry maps a human-readable string (e.g. "app.log") to an integer inode number.
  • Hard Links: A hard link adds another directory entry pointing to an existing inode number, incrementing i_nlink. Both names have identical status.
  • File Deletion Mechanics: When you run rm file, the kernel issues the unlink() system call. The kernel decrements i_nlink. If i_nlink reaches 0, the disk blocks are freed only after every open file descriptor referencing that inode is closed. If a running daemon holds the file descriptor open, the disk space remains allocated!

Ext4 Journaling Modes:

Ext4 uses a circular write-ahead log (the journal) to prevent filesystem corruption after power loss:

  1. data=journal: Both filesystem metadata and file contents are committed to the journal before touching the filesystem. Maximum integrity, but slowest write throughput (double writes).
  2. data=ordered (Default): File contents are written to their final disk blocks before the associated metadata is committed to the journal. Guarantees that unwritten garbage data is never exposed after an unexpected crash.
  3. data=writeback: Only metadata changes are journaled. File data blocks can be written before or after metadata. Maximum throughput, but crash recovery may leave old deleted data inside newly appended files.

4. The Block Layer & struct bio

The Block Layer handles data in units of hardware sectors and filesystem blocks.

The fundamental unit of block I/O in the Linux kernel is struct bio (defined in include/linux/bio.h):

struct bio {
    struct bio          *bi_next;       /* Request queue linkage */
    struct block_device *bi_bdev;       /* Target block device */
    blk_opf_t           bi_opf;         /* Operation flags: REQ_OP_READ, REQ_OP_WRITE */
    unsigned short      bi_flags;       /* Status flags */
    struct bvec_iter    bi_iter;        /* Current sector & bio_vec iterator */
    unsigned int        bi_vcnt;        /* Number of bio_vec elements */
    struct bio_vec      *bi_io_vec;     /* Array of (page, offset, len) vectors */
    bio_end_io_t        *bi_end_io;     /* Completion callback function */
};

Multi-Queue Block Layer (blk-mq)

On modern multi-core systems with high-speed NVMe drives, traditional single-lock request queues caused severe CPU contention. The modern Linux kernel uses blk-mq (Multi-Queue Block Layer):

  • Software Staging Queues: Per-CPU queues that eliminate cross-core lock contention.
  • Hardware Dispatch Queues: Maps 1:1 or N:M with physical NVMe hardware submission queues.

5. The Asynchronous I/O Evolution: Why io_uring?

For decades, network servers relied on epoll for non-blocking network sockets. However:

  1. epoll does not support regular files: Regular disk files always report as “ready” even when reading from cold physical disks will stall the calling thread on a major page fault.
  2. POSIX AIO (aio_read()): Historically implemented via helper threads in glibc, incurring thread synchronization overhead.
  3. Linux Kernel AIO (io_submit): Limited almost exclusively to O_DIRECT file access and cannot handle buffered files, network sockets, or pipes.

Enter io_uring

Designed in 2019 by Linux kernel block layer maintainer Jens Axboe, io_uring is a true high-performance asynchronous interface that works across any file descriptor: regular files, sockets, pipes, and devices.

It operates using two lockless ring buffers mapped into both user space and kernel space memory:

  1. Submission Queue (SQ): Ring buffer where user space enqueues I/O requests (struct io_uring_sqe).
  2. Completion Queue (CQ): Ring buffer where the kernel publishes completed I/O results (struct io_uring_cqe).
+--------------------------------------------------------------------------+
| USER SPACE MEMORY                                                        |
|                                                                          |
|   Submission Queue (SQ) Ring                Completion Queue (CQ) Ring   |
|   [SQE 0] [SQE 1] [SQE 2]                   [CQE 0] [CQE 1] [CQE 2]      |
|     |       |       |                         ^       ^       ^          |
+-----|-------|-------|-------------------------|-------|-------|----------+
      |       |       | (Shared via mmap)       |       |       |
+-----v-------v-------v-------------------------|-------|-------|----------+
| KERNEL SPACE                                  |       |       |          |
|                                               |       |       |          |
|   Worker Threads / SQPOLL Kernel Thread ------+-------+-------+          |
|   Dispatches bio requests to NVMe controller                             |
+--------------------------------------------------------------------------+

Because both queues reside in shared memory (mmap), submitting and completing requests requires zero memory copying and—with IORING_SETUP_SQPOLL—zero system calls.

6. Hands-On Practical Labs

Lab 7.1: Asynchronous File I/O with liburing in C

In this lab, we build a compilable C program that initializes an io_uring ring, submits an asynchronous vector read request for a file, and reaps the completion event without blocking.

Prerequisites

Ensure liburing-dev (Debian/Ubuntu) or liburing-devel (RHEL/Fedora) is installed:

sudo apt-get install -y liburing-dev build-essential

Program Source: uring_reader.c

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <string.h>
#include <unistd.h>
#include <liburing.h>

#define QUEUE_DEPTH 8
#define BLOCK_SZ    4096

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
        return 1;
    }

    const char *filepath = argv[1];
    int fd = open(filepath, O_RDONLY);
    if (fd < 0) {
        perror("open failed");
        return 1;
    }

    /* 1. Initialize io_uring with queue depth of 8 entries */
    struct io_uring ring;
    int ret = io_uring_queue_init(QUEUE_DEPTH, &ring, 0);
    if (ret < 0) {
        fprintf(stderr, "io_uring_queue_init failed: %s\n", strerror(-ret));
        close(fd);
        return 1;
    }

    /* 2. Allocate aligned memory buffer */
    char *buffer = malloc(BLOCK_SZ);
    if (!buffer) {
        perror("malloc failed");
        close(fd);
        io_uring_queue_exit(&ring);
        return 1;
    }

    /* 3. Fetch a Submission Queue Entry (SQE) */
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    if (!sqe) {
        fprintf(stderr, "Failed to get SQE from ring\n");
        goto cleanup;
    }

    /* 4. Prepare asynchronous read request (offset 0, size BLOCK_SZ) */
    io_uring_prep_read(sqe, fd, buffer, BLOCK_SZ - 1, 0);
    sqe->user_data = 1337; /* Identifier to match upon completion */

    /* 5. Submit SQE to kernel */
    printf("[+] Submitting async read request to kernel via io_uring...\n");
    io_uring_submit(&ring);

    /* 6. Wait for Completion Queue Entry (CQE) */
    struct io_uring_cqe *cqe;
    printf("[+] Waiting for completion event from CQ ring...\n");
    ret = io_uring_wait_cqe(&ring, &cqe);
    if (ret < 0) {
        fprintf(stderr, "io_uring_wait_cqe failed: %s\n", strerror(-ret));
        goto cleanup;
    }

    /* 7. Verify result */
    if (cqe->res < 0) {
        fprintf(stderr, "Async I/O failed with error: %s\n", strerror(-cqe->res));
    } else {
        buffer[cqe->res] = '\0';
        printf("[+] Read %d bytes successfully (user_data: %llu)!\n", cqe->res, (unsigned long long)cqe->user_data);
        printf("[+] Content Preview:\n---\n%s\n---\n", buffer);
    }

    /* 8. Mark CQE as consumed */
    io_uring_cqe_seen(&ring, cqe);

cleanup:
    free(buffer);
    close(fd);
    io_uring_queue_exit(&ring);
    return 0;
}

Compilation & Execution

# Compile with liburing
gcc -O2 uring_reader.c -o uring_reader -luring

# Execute against /etc/os-release
./uring_reader /etc/os-release

Lab 7.2: Benchmarking io_uring vs libaio vs psync with fio

The fio (Flexible I/O Tester) benchmark is the industry standard for measuring Linux storage throughput and latency.

1. Install fio

sudo apt-get install -y fio

2. Run Comparative Benchmark Harness

Create a test configuration benchmark.fio:

[global]
ioengine=io_uring
direct=1
runtime=15s
time_based=1
group_reporting=1
filename=/tmp/fio_test.dat
size=512M
bs=4k
rw=randread
numjobs=4
iodepth=64

[io_uring_test]
ioengine=io_uring

[libaio_test]
stonewall
ioengine=libaio

[psync_test]
stonewall
ioengine=psync
iodepth=1

Run the benchmark:

fio benchmark.fio

What to Observe:

  • IOPS: Notice how io_uring achieves significantly higher IOPS than psync due to asynchronous batching and zero per-request context switching.
  • CPU Utilization: io_uring consumes significantly fewer CPU cycles per I/O operation compared to POSIX libaio.

Lab 7.3: Investigating Page Cache Writeback & Dirty Pages

Inspect the kernel’s real-time Page Cache dirty page statistics:

# Display active dirty memory pages
grep -E 'Dirty|Writeback' /proc/meminfo

# Monitor disk I/O activity every second
vmstat -d 1

# Inspect active kernel dirty ratio settings
sysctl vm.dirty_ratio vm.dirty_background_ratio

To observe background flushing in real-time, write 1GB of random data:

dd if=/dev/urandom of=/tmp/test_dirty.bin bs=1M count=1000 &
while pgrep dd > /dev/null; do
    grep "Dirty:" /proc/meminfo
    sleep 0.5
done
rm -f /tmp/test_dirty.bin

Lab 7.4: Inode Lifecycle, Hard Links & Unlinked Open File Space Reclamation

A frequent production dilemma covered in How Linux Works is when a disk partition reports 100% full (df -h), but du -sh * fails to find where the storage went. This occurs when an unlinked (deleted) file remains open by a running background daemon.

# 1. Create a 200MB test file and check its inode and link count
dd if=/dev/zero of=/tmp/ghost_leak.bin bs=1M count=200 status=none
ls -l -i /tmp/ghost_leak.bin
# Notice link count = 1

# 2. Spawn a background process that holds the file open for 60 seconds
python3 -c "import time; f = open('/tmp/ghost_leak.bin', 'r'); time.sleep(60)" &
PY_PID=$!
echo "Process PID $PY_PID holds open file descriptor on /tmp/ghost_leak.bin"

# 3. Delete the file using rm (issues unlink syscall)
rm /tmp/ghost_leak.bin

# 4. Notice: 'ls' shows the file is gone!
ls -l /tmp/ghost_leak.bin 2>/dev/null || echo "File no longer visible in directory tree."

# 5. But disk blocks are NOT reclaimed because the kernel inode reference count > 0!
# Find the leaked open unlinked file using lsof:
lsof +L1 | grep "ghost_leak"

# 6. Inspect the file directly via the process file descriptor symlink in /proc:
ls -l /proc/$PY_PID/fd/

# 7. Terminate the process holding the descriptor:
kill -9 $PY_PID
# Now the kernel reclaims the physical disk blocks immediately!

7. Summary Checklist

  • VFS Abstraction: Understand struct file, struct inode, and struct dentry.
  • Filesystem Anatomy: Mastered how inodes store metadata without filenames, and how directories act as name-to-inode tables.
  • Ext4 Journaling: Evaluated trade-offs between data=journal, data=ordered, and data=writeback.
  • Page Cache & Direct I/O: Mastered O_DIRECT alignment rules and vm.dirty_* sysctl controls.
  • Block Layer: Explored struct bio and blk-mq multi-queue dispatch.
  • io_uring: Understood Submission Queues (SQ) and Completion Queues (CQ) mapped via shared memory.
  • Superuser Troubleshooting: Diagnosed ghost unlinked disk space leaks using lsof +L1 and /proc/<PID>/fd/.
  • Practical Hands-on: Compiled and executed a real asynchronous io_uring reader in C.