Process & Thread Management

“There are no ’threads’ in the Linux kernel. There are only tasks that happen to share an address space and file descriptors, and tasks that do not. The distinction between a process and a thread is a user-space abstraction.”
— Linus Torvalds

In traditional operating system textbooks, a Process is taught as an isolated program execution unit, while a Thread is taught as a lightweight execution unit inside a process.

In the Linux kernel, this distinction does not exist. Both processes and threads are represented by the exact same internal C structure: struct task_struct.

Understanding how the Linux kernel manages tasks, executes context switches, clones execution contexts via Copy-On-Write, and enforces resource isolation is essential for systems engineering, database optimization, and cloud-native container runtimes.

1. The Core Data Structure: struct task_struct

Every execution context in Linux is tracked by a process descriptor: the task_struct (defined in include/linux/sched.h). It is one of the largest and most critical structures in the kernel, holding hundreds of fields:

struct task_struct {
    /* 1. Identification */
    pid_t               pid;            /* Thread ID (TID) in user space */
    pid_t               tgid;           /* Thread Group ID (Process ID in user space) */
    
    /* 2. Process State */
    unsigned int        __state;        /* TASK_RUNNING, TASK_INTERRUPTIBLE, etc. */
    int                 exit_state;     /* EXIT_ZOMBIE, EXIT_DEAD */
    int                 exit_code;      /* Exit status returned to parent */

    /* 3. Memory & Virtual Address Space */
    struct mm_struct    *mm;            /* Memory descriptor (NULL for kernel threads!) */
    struct mm_struct    *active_mm;     /* Borrowed mm for kernel threads (lazy TLB) */

    /* 4. Filesystem & Descriptors */
    struct fs_struct    *fs;            /* Filesystem info (root dir, cwd, umask) */
    struct files_struct *files;         /* File descriptor table (open fds) */

    /* 5. Scheduling & Runqueues */
    int                 prio;           /* Dynamic priority */
    int                 static_prio;    /* Nice value (-20 to +19) */
    struct sched_entity se;             /* EEVDF / CFS scheduling entity (vruntime) */

    /* 6. Process Relationships & Tree Hierarchy */
    struct task_struct  __rcu *real_parent; /* Real parent process */
    struct list_head    children;       /* List of child processes */
    struct list_head    sibling;        /* Linkage in parent's children list */

    /* 7. Signal Handling */
    struct signal_struct *signal;       /* Shared signal handlers */
    sigset_t            blocked;        /* Mask of currently blocked signals */
};

PID vs. TGID: The POSIX Illusion

In standard user space POSIX programming:

• When you invoke getpid(), you expect your Process ID.
• In multi-threaded programs (e.g., using pthread_create), all threads share the same Process ID.

Under the hood:

• The Linux kernel assigns a completely unique pid to every single thread and process.
• To satisfy POSIX, the kernel introduces the Thread Group ID (tgid).
• For the main process thread, pid == tgid. When secondary threads are spawned, they receive new, unique pid values, but inherit the main process’s tgid.
• In Linux:
  • getpid() system call returns current->tgid.
  • gettid() system call returns current->pid.

2. Process Creation: fork, clone, and Copy-On-Write (COW)

Linux does not create new processes from scratch; it clones existing ones using the clone() or clone3() system call.

       Parent Process (PID: 1000)
                  |
                  |  Calls fork() / clone()
                  v
       Kernel duplicates task_struct
       Duplicates Page Tables (Marks physical frames as READ-ONLY / COW)
                  |
         +--------+--------+
         |                 |
         v                 v
   Parent (PID 1000)   Child (PID 1001)
   returns child PID   returns 0
         |                 |
         v                 v
  [ Both read identical physical memory frames with ZERO copying ]

1. Copy-On-Write (COW) Optimization

Historically in Unix, fork() physically copied the entire memory space of the parent into the child. If a process consumed 10 Gigabytes of RAM, fork() would duplicate 10 Gigabytes, incurring catastrophic allocation latency.

Linux uses Copy-On-Write (COW):

1. When fork() is called, the kernel duplicates the parent’s page table entries, pointing the child’s virtual addresses to the exact same physical RAM frames as the parent.
2. The kernel marks all shared physical pages as Read-Only and flags them as COW in the MMU.
3. Both parent and child continue executing, reading from shared memory at zero cost.
4. The Write Fault: The moment either process attempts to write to a memory page (e.g., modifying a variable), the MMU hardware triggers a Page Fault (#PF).
5. The kernel intercepts the fault in do_wp_page():
   • It allocates a single, fresh 4KB physical page from the Buddy Allocator.
   • It copies the 4KB data from the shared frame into the new frame.
   • It updates the writing process’s page table entry with write permissions.
   • It leaves all other memory pages untouched and shared.

2. The clone() Flags: How Threads are Born

When creating a task via clone(), flags control what is shared versus what is copied:

// Creating a lightweight Thread (shared address space and files):
clone(fn, stack, CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, arg);

// Creating an isolated Process:
clone(fn, stack, SIGCHLD, arg);

// Creating an isolated Container Process:
clone(fn, stack, CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | CLONE_NEWIPC | SIGCHLD, arg);

3. The execve() System Call: Binary Transformation

While fork() creates an identical duplicate of a process, execve() wipes away the existing program and replaces it with a completely new executable binary.

The Execution Pipeline:

1. Validating the Binary: The kernel opens the executable file and inspects the first bytes for the ELF Magic Number (0x7F 'E' 'L' 'F') or a script hashbang (#!/bin/bash).
2. Releasing Old Memory: The kernel destroys the process’s existing mm_struct, unmapping all old code, heap, and libraries.
3. Setting Up Address Space: The kernel parses the ELF headers and creates new Virtual Memory Areas (VMAs):
   • .text segment: Executable code (Read/Execute).
   • .data segment: Initialized global variables (Read/Write).
   • .rodata segment: Constants and string literals (Read-Only).
   • .bss segment: Zero-initialized variables (Anonymous pages).
4. Setting Up User Stack: The kernel pushes command-line arguments (argv), environment variables (envp), and the Auxiliary Vector (auxv) onto the new user stack.
5. Jumping to Entry Point: The kernel sets the instruction pointer (%rip on x86-64) to the binary’s entry address (or to the dynamic linker /lib64/ld-linux-x86-64.so.2) and transitions into Ring 3.

4. Process State Machine & The Zombie Lifecycle

A Linux task traverses through well-defined lifecycle states:

               +-------------------+
               |    TASK_NEW       |
               +-------------------+
                         |
                         v
   +-------------> [TASK_RUNNING] <-------------+
   |             (Ready on Runqueue             |
   |              or executing on CPU)          |
   |                     |                      |
Scheduler           I/O Block /              I/O Complete /
Preempt             Wait Event               Signal Received
   |                     |                      |
   |                     v                      |
   |           [TASK_INTERRUPTIBLE] ------------+
   |                     |
   |                     | (Uninterruptible Disk / Lock Sleep)
   |                     v
   |          [TASK_UNINTERRUPTIBLE (D)]
   |                     |
   |                     v
   |             [TASK_STOPPED (T)] (via SIGSTOP / Debugger)
   |                     |
   +---------------------+
                         |
                   exit() called
                         v
                [EXIT_ZOMBIE (Z)]
                         |
                Parent calls waitpid()
                         v
                 [EXIT_DEAD (X)]
               (task_struct freed)

The Truth About Zombie Processes (EXIT_ZOMBIE):

A common interview question: What is a zombie process, and how do you kill it?

• A zombie process is already dead. It consumes zero CPU, zero RAM, and holds no open file descriptors.
• Why does it exist? When a child calls exit(), the kernel frees its memory and resources, but retains its task_struct in the process table to hold its exit code (e.g., exit(0) vs exit(1)).
• The parent process must read this exit code by calling wait() or waitpid(). Once the parent calls wait(), the zombie is reaped and its task_struct is removed.
• You cannot kill a zombie with SIGKILL (kill -9) because a zombie is already dead and cannot execute a signal handler. To eliminate a zombie, you must either fix or terminate its parent process, which causes the zombie to be reparented to PID 1 (systemd), which immediately calls wait() to reap it.

5. The Context Switch: What Happens on Silicon?

When the scheduler decides to stop Task A and run Task B, the kernel executes a Context Switch:

[ CPU Core: Running Task A in Ring 3 ]
                  |
                  | 1. Hardware Timer Interrupt (Local APIC) fires
                  v
[ CPU enters Ring 0: irq_enter() ]
                  |
                  | 2. scheduler_tick() updates task vruntime
                  | 3. Detects Task A has exhausted timeslice
                  | 4. Sets TIF_NEED_RESCHED flag on Task A
                  v
[ schedule() -> __schedule() ]
                  |
                  | 5. EEVDF scheduler selects Task B as next task
                  v
[ context_switch(prev=Task A, next=Task B) ]
                  |
                  +---> A. switch_mm() (Memory Space Switch)
                  |        Loads Task B's page table physical address into CR3 register.
                  |        (Flushes TLB unless PCID / ASID tags are supported).
                  |        *Skipped if Task B is a kernel thread! (Lazy TLB)*
                  |
                  +---> B. switch_to() (Register & Stack Switch)
                           Saves Task A's CPU registers (rsp, rbp, rbx, r12-r15)
                           Restores Task B's saved registers from its kernel stack.
                           Switches CPU Stack Pointer (%rsp) to Task B's stack.
                  v
[ CPU resumes in Ring 3: Task B continues executing ]

6. Signals, Priorities & /proc/<PID>/ Superuser Anatomy

In How Linux Works, Brian Ward emphasizes that signals and the /proc filesystem are the two primary tools superusers possess to inspect and control processes.

1. Unix Signals: Asynchronous Process Interrupts

Signals notify a process that an event has occurred. The kernel checks for pending signals whenever a process transitions from kernel mode back to user mode.

2. Process Priorities & Nice Values

Linux processes execute with priorities between 0 and 139:

• Real-Time Priorities (0 to 99): Controlled by chrt using SCHED_FIFO or SCHED_RR.
• Standard Nice Values (-20 to +19): Maps to kernel static priorities 100 to 139.
  • Default nice is 0.
  • -20: Highest priority (least “nice” to other tasks; gets largest CPU slice).
  • +19: Lowest priority (most “nice”; yields to everything else).
  • Regular users can only increase their nice value (reduce priority). Only root can assign negative nice values.

# Launch background compilation with low CPU priority
nice -n 19 make -j4

# Increase priority of a running production database (root required)
sudo renice -n -10 -p 1842

3. The /proc/<PID>/ Anatomy

The /proc filesystem directly exposes the kernel’s internal task_struct for every running PID:

• /proc/<PID>/cmdline: Exact command-line arguments separated by null bytes (\0).
• /proc/<PID>/environ: Process environment variables.
• /proc/<PID>/fd/: Directory containing symlinks for every open file descriptor (0=stdin, 1=stdout, 2=stderr, sockets, open disk files).
• /proc/<PID>/status: Human-readable process state:
  • State: (e.g., S (sleeping), D (disk sleep), Z (zombie)).
  • VmPeak & VmSize: Total virtual memory mapped.
  • VmRSS: Resident Set Size (actual physical RAM frames consumed).
  • voluntary_ctxt_switches: Number of times the task slept waiting for I/O.
• /proc/<PID>/limits: Soft and hard resource limits (RLIMIT_NOFILE max open files, RLIMIT_NPROC).
• /proc/<PID>/wchan: The exact kernel function where the process is currently blocked waiting (invaluable for diagnosing stuck processes in D state!).

7. Modern Container Primitives: Namespaces & cgroups

Containers (Docker, Podman, Kubernetes, containerd) are not hardware virtual machines. A container is simply a standard Linux process isolated by Namespaces and restricted by Control Groups (cgroups).

+------------------------------------------------------------------------+
|                          WHAT IS A CONTAINER?                          |
|                                                                        |
|    Standard Linux Process (task_struct)                                |
|    +                                                                   |
|    Linux Namespaces (What the process can SEE: Isolated Views)         |
|    +                                                                   |
|    Linux Control Groups (What the process can USE: Resource Limits)    |
+------------------------------------------------------------------------+

1. Linux Namespaces (Isolation)

Namespaces partition global system resources so that a group of processes sees an isolated instance:

2. Control Groups (cgroups v2: Resource Limiting)

While namespaces control visibility, cgroups govern resource consumption to prevent noisy neighbors from exhausting host capacity:

• cpu.max: Controls CPU bandwidth quota via CFS/EEVDF period throttling (e.g., 200000 100000 = 2 dedicated CPU cores).
• memory.max: Hard physical RAM ceiling. Exceeding this boundary triggers the cgroup OOM Killer.
• io.weight / io.max: Throttles read/write IOPS and bytes-per-second on block devices.
• pids.max: Limits maximum fork capacity to prevent fork-bomb denial of service attacks.

# Inspect the cgroups v2 tree of any active process on your machine:
cat /proc/self/cgroup

# Inspect active namespaces:
ls -l /proc/self/ns/

8. Hands-On Practical Labs

Lab 6.1: Creating, Inspecting, and Reaping Zombie Processes in C

A zombie process occurs when a child process terminates, but its parent has not yet read its exit status code via wait() or waitpid(). The kernel frees the child’s memory pages but retains its struct task_struct and PID slot.

Complete Source: zombie_demo.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main(void) {
    pid_t pid = fork();

    if (pid < 0) {
        perror("fork failed");
        return 1;
    }

    if (pid == 0) {
        /* Child Process: Terminates immediately */
        printf("[Child: PID %d] Exiting now with code 42...\n", getpid());
        exit(42);
    } else {
        /* Parent Process: Sleeps without calling wait() */
        printf("[Parent: PID %d] Child spawned (PID %d).\n", getpid(), pid);
        printf("[Parent] Sleeping 15 seconds WITHOUT reaping child. Inspect ps aux in another terminal!\n");
        sleep(15);

        printf("[Parent] Waking up. Now calling waitpid() to reap child...\n");
        int status;
        waitpid(pid, &status, 0);

        if (WIFEXITED(status)) {
            printf("[Parent] Successfully reaped child! Exit code was: %d\n", WEXITSTATUS(status));
        }

        printf("[Parent] Sleeping 5 more seconds. Verify child is completely gone from ps!\n");
        sleep(5);
    }

    return 0;
}

Verification Steps:

# 1. Compile and run
gcc -O2 zombie_demo.c -o zombie_demo
./zombie_demo &

# 2. In another terminal, inspect process state while parent sleeps:
ps aux | grep -E 'defunct|zombie_demo'

Output:

user  18400  0.0  0.0   2488   812 pts/0  S+   12:00   0:00 ./zombie_demo
user  18401  0.0  0.0      0     0 pts/0  Z+   12:00   0:00 [zombie_demo] <defunct>

Notice:

• The child PID has status Z+ (<defunct>) and 0 KB memory.
• Running kill -9 18401 does nothing! The process is already dead; only its task descriptor remains until the parent calls waitpid() or dies.

Lab 6.2: Tracing clone() System Call Flags with strace

In this lab, we observe how the Linux kernel uses clone() under the hood when spawning POSIX threads.

1. Multithreaded Program: thread_test.c

#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

void *worker(void *arg) {
    printf("[Worker] Thread active. TID: %ld\n", (long)gettid());
    sleep(1);
    return NULL;
}

int main(void) {
    pthread_t t1;
    printf("[Main] PID: %ld, TID: %ld\n", (long)getpid(), (long)gettid());
    pthread_create(&t1, NULL, worker, NULL);
    pthread_join(t1, NULL);
    return 0;
}

2. Trace with strace:

gcc -pthread thread_test.c -o thread_test
strace -f -e clone,clone3 ./thread_test

Expected Kernel Output:

clone(child_stack=0x7f4b821fedf0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, ...)

What This Proves:

• CLONE_VM: Shares virtual address space (memory).
• CLONE_FILES: Shares the open file descriptor table.
• CLONE_THREAD: Assigns the same tgid (Thread Group ID).

Lab 6.3: Handling Asynchronous Signals (sigaction) & Auditing /proc/<PID>/status

In How Linux Works, superusers learn that daemons should catch SIGHUP to reload config without restarting. Here, we build a daemon in C that handles SIGHUP and SIGTERM, and inspect its /proc state.

1. Source Code: signal_daemon.c

#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <unistd.h>

static volatile sig_atomic_t reload_requested = 0;
static volatile sig_atomic_t keep_running = 1;

void handle_signal(int sig) {
    if (sig == SIGHUP) {
        reload_requested = 1;
    } else if (sig == SIGTERM || sig == SIGINT) {
        keep_running = 0;
    }
}

int main(void) {
    struct sigaction sa = {0};
    sa.sa_handler = handle_signal;
    sigemptyset(&sa.sa_mask);

    sigaction(SIGHUP, &sa, NULL);
    sigaction(SIGTERM, &sa, NULL);
    sigaction(SIGINT, &sa, NULL);

    printf("[Daemon] Started. PID: %d. Waiting for signals...\n", getpid());
    int tick = 0;

    while (keep_running) {
        sleep(1);
        tick++;

        if (reload_requested) {
            printf("[Daemon: PID %d] SIGHUP received! Reloading configuration dynamically...\n", getpid());
            reload_requested = 0;
        }
    }

    printf("[Daemon: PID %d] SIGTERM received! Flushed buffers, exiting cleanly.\n", getpid());
    return 0;
}

2. Verification Steps:

# 1. Compile and run in background
gcc -O2 signal_daemon.c -o signal_daemon
./signal_daemon &
DPID=$!

# 2. Inspect caught signals mask in /proc
grep -E 'SigCgt|SigIgn|State' /proc/$DPID/status

# 3. Send SIGHUP to trigger configuration reload
kill -HUP $DPID

# 4. Gracefully terminate with SIGTERM
kill -TERM $DPID
wait $DPID 2>/dev/null

9. Summary Checklist

• No Threads Myth: Understood that both processes and threads are task_struct instances.
• PID vs TGID: Mastered how POSIX compatibility is maintained inside the kernel.
• Copy-On-Write: Traced page table duplication and write-fault traps.
• Signals Architecture: Mastered SIGTERM vs SIGKILL, signal masks, and safe sigaction handling.
• Process Priorities: Configured nice values (-20 to +19) and real-time scheduling.
• /proc/<PID>/ Diagnostics: Audited memory RSS, file descriptors (/proc/PID/fd), and blocked functions (wchan).
• Context Switching: Dissected switch_mm (CR3 reload) and switch_to register swapping.
• Hands-on Labs: Reaped zombie processes, traced clone() flags, and handled POSIX signals in C.