Container Primitives: Namespaces, cgroups v2 & OverlayFS

“Containers are an illusion. There is no such thing as a ‘container’ object in the Linux kernel. A container is simply a standard Linux process that has been isolated using Namespaces and throttled using Control Groups.”
— Linus Torvalds

Engineers often discuss Docker, Podman, and Kubernetes as if they were virtualization hypervisors like VMware or KVM. In reality, every single container running in a production Kubernetes cluster is an ordinary Linux process running directly on the host kernel.

What makes a process a “container” is the clever combination of three core Linux kernel technologies:

1. Namespaces (Virtualizing what a process can SEE).
2. Control Groups / cgroups v2 (Restricting what a process can USE).
3. Union Filesystems / OverlayFS & pivot_root (Isolating the FILESYSTEM ROOT).

In this guide, we demystify container engines by building a container runtime from scratch using raw Linux kernel primitives.

1. Virtual Machines vs. Containers: The Virtualization Spectrum

In How Linux Works, Brian Ward explores virtualization by contrasting full hardware virtualization with lightweight OS-level container isolation:

    HARDWARE VIRTUALIZATION (KVM / QEMU)             OS-LEVEL CONTAINERS (Docker / Podman)
+------------------------------------------+    +------------------------------------------+
|  Guest App    |  Guest App   | Guest App |    |  App 1       |  App 2       |  App 3     |
+---------------+--------------+-----------+    +--------------+--------------+------------+
|  GUEST OS (Independent Linux / Windows)  |    |  Isolated Namespaces & cgroups v2 limits  |
+------------------------------------------+    +------------------------------------------+
|  VIRTUAL HARDWARE (Emulated by QEMU)     |    |                                          |
+------------------------------------------+    |                                          |
|  HYPERVISOR: KVM (/dev/kvm) in Linux     |    |           SINGLE HOST LINUX KERNEL       |
+------------------------------------------+    |        (Shared Ring 0 Address Space)     |
|           PHYSICAL HARDWARE              |    |                                          |
+------------------------------------------+    +------------------------------------------+

The Architectural Trade-offs:

2. The Anatomy of a Container

A container is not a physical boundary; it is a standard Linux process isolated by Namespaces and constrained by Control Groups:

+=============================================================================+
|                               HOST LINUX KERNEL                             |
|  Single Shared Kernel: Memory Manager, Scheduler, VFS, Drivers (Ring 0)     |
+=============================================================================+
        |                                                 |
        v                                                 v
+-------------------------------+       +------------------------------------+
| HOST PROCESSES                |       | "CONTAINERIZED" PROCESS            |
|                               |       |                                    |
| PID: 1 (systemd)              |       | Host PID: 18290                    |
| PID: 842 (dockerd)            |       | Container PID: 1 (via CLONE_NEWPID)|
|                               |       |                                    |
| View: Global host filesystem  |       | View: OverlayFS mount (via pivot_root)
| Network: Host eth0 / 10.0.0.5 |       | Network: veth0 / 172.17.0.2        |
| Resource: Uncapped CPU/RAM    |       | Resource: Capped at 512MB RAM      |
| Capabilities: Full root       |       | Capabilities: Dropped CAP_SYS_ADMIN|
+-------------------------------+       +------------------------------------+

3. The 7 Linux Namespaces

Namespaces partition global system resources so that a group of processes observes an independent, virtualized instance. When a process inspects its namespaces, it can view them in /proc/[PID]/ns/:

ls -l /proc/self/ns/

4. Resource Limiting: Control Groups v2 (cgroups v2)

While namespaces isolate visibility, Control Groups (cgroups) constrain resource consumption to prevent runaway processes from starving host resources.

In Linux 4.5+, cgroups v2 introduced a single unified hierarchy mounted at /sys/fs/cgroup. Every process belongs to exactly one cgroup in the tree.

Core Controllers in cgroups v2:

• memory.max: Hard physical memory boundary. When memory usage exceeds this value, the kernel attempts page reclamation. If memory remains exhausted, the kernel cgroup OOM Killer terminates processes inside the cgroup.
• cpu.max: Bandwidth limit defined as $QUOTA $PERIOD in microseconds. For example, 200000 100000 allows the cgroup to consume at most 200ms of CPU time per 100ms wall-clock interval (equivalent to 2.0 CPU cores).
• io.max: Throttles read and write IOPS and bytes-per-second on specified block devices (e.g., 8:0 rbps=10485760 limits major:minor device 8:0 to 10MB/s read).
• pids.max: Caps the number of active tasks/threads to protect against fork-bombs.

5. Layered Filesystems: OverlayFS & pivot_root

Containers achieve instant startup times and low disk footprints through OverlayFS, a union mount filesystem:

+--------------------------------------------------------------------------+
| MERGED VIEW (/merged) -> What the running container sees                 |
+--------------------------------------------------------------------------+
       ^                                            ^
       | (Modifications go here)                    | (Read-only base layers)
+----------------------------------+   +------------------------------------+
| UPPER DIR (/upper)               |   | LOWER DIR (/lower - e.g., Ubuntu)  |
| Read-Write container layer       |   | Immutable read-only Docker image   |
+----------------------------------+   +------------------------------------+

pivot_root vs. chroot

Historically, programs used chroot(). However, chroot() does not truly isolate the filesystem—a privileged process can escape a chroot jail via relative path traversing (../../).

Containers use pivot_root():

• It swaps the root filesystem of the calling process with a new directory.
• It moves the old host root to a temporary sub-directory, which is then unmounted with umount -l /old_root and deleted.
• Escaping a properly configured pivot_root mount namespace is cryptographically and architecturally impossible without breaking kernel memory isolation.

6. Hands-On Practical Labs

Lab 8.1: Building a Complete Container from Scratch in Shell

In this lab, we construct an isolated container runtime using raw Linux command-line utilities without Docker or containerd.

Step 1: Create a Minimal Rootfs

Download and extract an Alpine Linux root filesystem:

mkdir -p /tmp/mycontainer/rootfs
cd /tmp/mycontainer

# Download minimal Alpine rootfs tarball
curl -sSL -o alpine.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.0-x86_64.tar.gz
tar -xzf alpine.tar.gz -C rootfs/

Step 2: Write the Container Launch Script: run_container.sh

#!/bin/bash
set -e

CONTAINER_DIR="/tmp/mycontainer"
ROOTFS="$CONTAINER_DIR/rootfs"

echo "[*] Step 1: Configuring Mount Namespace and Private Propagation..."
# Unshare namespaces: PID, Mount, UTS, IPC, Network
exec unshare --mount --uts --ipc --pid --net --fork /bin/bash << 'EOF'

# 1. Set hostname inside container UTS namespace
hostname my-isolated-container

# 2. Prevent mount events from leaking to host
mount --make-rprivate /

# 3. Mount proc inside container rootfs
mkdir -p /tmp/mycontainer/rootfs/proc
mount -t proc proc /tmp/mycontainer/rootfs/proc

# 4. Mount sysfs inside container rootfs
mkdir -p /tmp/mycontainer/rootfs/sys
mount -t sysfs sys /tmp/mycontainer/rootfs/sys

# 5. Pivot root into the new rootfs
cd /tmp/mycontainer/rootfs
mkdir -p old_root
pivot_root . old_root

# 6. Unmount old host root
umount -l /old_root
rmdir /old_root

# 7. Launch isolated shell as PID 1
echo "[+] Successfully entered isolated container environment!"
echo "[+] Hostname: $(hostname)"
echo "[+] Active processes inside container:"
ps aux

# Start interactive shell
exec /bin/sh
EOF

Step 3: Run and Verify Isolation

chmod +x run_container.sh
sudo ./run_container.sh

Inside the container shell, verify:

• Run ps aux: You will see only /bin/sh running as PID 1!
• Run hostname: Shows my-isolated-container.
• Run ls /: Shows only the Alpine root filesystem.

Lab 8.2: Enforcing Memory Limits & Triggering cgroups v2 OOM Killer

In this lab, we restrict a process using pure cgroups v2 and observe the kernel terminate it when it exceeds memory thresholds.

Step 1: Verify cgroups v2 Mount

mount -t cgroup2
# Should output: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Step 2: Create a Sandbox cgroup Slice

sudo mkdir /sys/fs/cgroup/sandbox_demo

# Set memory ceiling to 50 Megabytes
echo "50M" | sudo tee /sys/fs/cgroup/sandbox_demo/memory.max

# Ensure swap does not mask the limit
echo "0" | sudo tee /sys/fs/cgroup/sandbox_demo/memory.swap.max

Step 3: Write a Python Memory Allocator: alloc_test.py

import sys
import time

print("[+] Allocating memory in 10MB chunks...")
chunks = []
try:
    for i in range(1, 10):
        print(f"Allocating chunk {i}: {i * 10} MB total")
        # Allocate 10MB and write to it to force physical page allocation
        chunk = bytearray(10 * 1024 * 1024)
        chunks.append(chunk)
        time.sleep(0.5)
except Exception as e:
    print(f"Exception: {e}")

Step 4: Run Allocator Inside cgroup

Attach the subshell to the cgroup and execute:

# Add current shell PID to cgroup tasks
echo $$ | sudo tee /sys/fs/cgroup/sandbox_demo/cgroup.procs

# Run allocator (will exceed 50MB and be terminated!)
python3 alloc_test.py

Step 5: Verify the OOM Event

Check kernel ring buffer logs:

dmesg | tail -n 25 | grep -E 'oom_reaper|Killed process|Memory cgroup out of memory'

Inspect cgroup v2 event counters:

cat /sys/fs/cgroup/sandbox_demo/memory.events

Output will confirm:

low 0
high 0
max 1
oom 1
oom_kill 1

Lab 8.3: System Call Sandboxing with Seccomp-BPF in C

Seccomp (Secure Computing Mode) uses the in-kernel BPF bytecode engine to filter system calls before they are executed. If a sandboxed process invokes a forbidden syscall, the kernel immediately halts or kills it with SIGSYS.

Source Code: seccomp_filter.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/prctl.h>
#include <linux/seccomp.h>
#include <linux/filter.h>
#include <linux/audit.h>
#include <sys/syscall.h>

void install_seccomp_filter(void) {
    /* 
     * Seccomp BPF Filter:
     * 1. Validate architecture (x86_64)
     * 2. Inspect syscall number
     * 3. If syscall is __NR_mkdir, KILL the process!
     * 4. Otherwise, ALLOW all other syscalls.
     */
    struct sock_filter filter[] = {
        /* [0] Load architecture into accumulator */
        BPF_STMT(BPF_LD | BPF_W | BPF_ABS, (offsetof(struct seccomp_data, arch))),
        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AUDIT_ARCH_X86_64, 1, 0),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL),

        /* [1] Load syscall number into accumulator */
        BPF_STMT(BPF_LD | BPF_W | BPF_ABS, (offsetof(struct seccomp_data, nr))),
        
        /* [2] If syscall is mkdir (sys_mkdir), KILL */
        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_mkdir, 0, 1),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),

        /* [3] Allow all other system calls */
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW)
    };

    struct sock_fprog prog = {
        .len = (unsigned short)(sizeof(filter) / sizeof(filter[0])),
        .filter = filter,
    };

    /* Prohibit child processes from regaining privileges via execve */
    if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) < 0) {
        perror("prctl(PR_SET_NO_NEW_PRIVS) failed");
        exit(1);
    }

    /* Install seccomp filter */
    if (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog) < 0) {
        perror("prctl(PR_SET_SECCOMP) failed");
        exit(1);
    }
}

int main(void) {
    printf("[+] Installing Seccomp-BPF policy to block mkdir syscall...\n");
    install_seccomp_filter();
    printf("[+] Policy active. Attempting allowed syscall: write()... SUCCESS!\n");

    printf("[+] Attempting FORBIDDEN syscall: mkdir('test_dir')...\n");
    fflush(stdout);

    /* This will be trapped and killed immediately by the kernel */
    syscall(__NR_mkdir, "test_dir", 0755);

    printf("[-] This line will NEVER be reached!\n");
    return 0;
}

Compilation & Execution

gcc -O2 seccomp_filter.c -o seccomp_filter
./seccomp_filter

Output:

[+] Installing Seccomp-BPF policy to block mkdir syscall...
[+] Policy active. Attempting allowed syscall: write()... SUCCESS!
[+] Attempting FORBIDDEN syscall: mkdir('test_dir')...
Bad system call (core dumped)

7. Summary Checklist

• Virtualization vs Containers: Contrasted hardware KVM/QEMU hypervisors with shared-kernel OS containers.
• Namespaces Demystified: Understood the 7 Linux namespaces and verified them in /proc/self/ns/.
• cgroups v2 Mastery: Created custom cgroup slices, set memory.max, and inspected memory.events.
• OverlayFS & pivot_root: Contrasted chroot with pivot_root for secure filesystem root isolation.
• Built a Container: Assembled a fully isolated runtime using standard shell and kernel primitives.
• Syscall Filtering: Implemented a real Seccomp-BPF filter in C.