Linux Foundations, Shell Mastery & systemd

“The Unix philosophy is simple: write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”
— Doug McIlroy

Before descending into Ring 0 kernel internals and memory page tables, every systems engineer must master user space: the Filesystem Hierarchy Standard (FHS), character and block devices (/dev), udev rule orchestration, Logical Volume Management (LVM), user authentication via PAM, and hardened systemd service sandboxing.

As Brian Ward emphasizes in How Linux Works: What Every Superuser Should Know, a superuser is not merely someone who knows commands—it is someone who understands how user space coordinates with the Linux kernel to manage hardware, processes, and storage.

1. Filesystem Hierarchy Standard (FHS)

Linux organizes files according to purpose, permanence, and sharing across systems:

/ (Root Directory)
├── etc/         Host-specific static configuration files (fstab, passwd, hosts)
├── bin, sbin    Symlinks to /usr/bin and /usr/sbin in modern systemd distros
├── usr/         Read-only software packages, libraries (/usr/lib), and headers
├── var/         Variable runtime state, persistent spool directories, /var/log
├── tmp/         Temporary files cleared upon reboot (often mounted as tmpfs)
├── dev/         Device nodes created dynamically by systemd-udevd (devtmpfs)
├── proc/        Virtual pseudo-filesystem exposing kernel data & task structures
└── sys/         Sysfs pseudo-filesystem exposing unified kernel device model

• /etc: Configuration files only. Executable binaries should never reside here.
• /var: Dynamic data written while the system runs: logs (/var/log), package caches (/var/cache), databases (/var/lib).
• /proc & /sys: Pure in-memory pseudo-filesystems. They consume 0 bytes of disk storage; reading a file in /proc or /sys triggers kernel C code to inspect internal data structures.

2. Devices, /dev, and Dynamic Management with udev

In Unix and Linux, devices are accessed as special files located under /dev. However, unlike regular files, device files do not contain data on disk; their inodes store a Major Number and a Minor Number.

$ ls -l /dev/sda /dev/tty0 /dev/urandom
brw-rw---- 1 root disk    8,  0 Oct 14 00:00 /dev/sda
crw--w---- 1 root tty     4,  0 Oct 14 00:00 /dev/tty0
crw-rw-rw- 1 root root    1,  9 Oct 14 00:00 /dev/urandom

Character vs. Block Devices:

• Character Devices (c): Represent streams of bytes that can be read or written sequentially without buffering by the block layer. Examples: terminals (/dev/tty*), serial ports, sound cards, and random number generators (/dev/urandom).
• Block Devices (b): Represent random-access storage media organized into fixed-size blocks (sectors). Block devices interact with the kernel’s Page Cache and I/O scheduler. Examples: NVMe drives (/dev/nvme0n1), SATA SSDs (/dev/sda), and loopback devices (/dev/loop0).

Major and Minor Numbers:

• Major Number: Identifies which kernel device driver handles operations for the device (e.g., Major 8 = SCSI/SATA disk driver, Major 1 = memory/raw drivers, Major 259 = NVMe block driver).
• Minor Number: Passed to the driver to identify the specific physical device or partition instance (e.g., /dev/sda has Minor 0, while /dev/sda1 has Minor 1).

The udev System (systemd-udevd):

In modern Linux, /dev is mounted as devtmpfs (a kernel-managed in-memory filesystem). When the kernel detects new hardware (USB plugged in, NVMe identified):

1. The kernel emits a uevent netlink message.
2. The user-space daemon systemd-udevd receives the event.
3. udev evaluates rules in /etc/udev/rules.d/ and /usr/lib/udev/rules.d/.
4. It sets device permissions, triggers helper scripts, and creates persistent descriptive symlinks under /dev/disk/by-uuid/, /dev/disk/by-id/, and /dev/disk/by-path/.

3. Disks, Partitions & Logical Volume Management (LVM)

MBR vs. GPT Partitioning:

• MBR (Master Boot Record): Legacy standard. Limited to 4 primary partitions and a maximum disk capacity of 2 Terabytes.
• GPT (GUID Partition Table): Modern standard required for UEFI booting. Supports 128+ partitions, 64-bit block addressing (up to 9.4 Zettabytes), and includes a secondary backup partition table at the end of the disk for disaster recovery.

The 3-Tier LVM Architecture:

Traditional raw disk partitions are rigid. If /var runs out of disk space, resizing an adjacent partition requires unmounting and moving data. LVM (Logical Volume Manager) inserts a virtualization layer between physical storage and filesystems:

+-------------------------------------------------------------------------+
|                  FILESYSTEMS (ext4, XFS, btrfs)                         |
+-------------------------------------------------------------------------+
                                    |
+-------------------------------------------------------------------------+
|     LOGICAL VOLUMES (LV): /dev/mapper/vg0-data, /dev/mapper/vg0-var     |
|     (Can be dynamically resized, snapshotted, or striped at runtime)    |
+-------------------------------------------------------------------------+
                                    ^
+-------------------------------------------------------------------------+
|                       VOLUME GROUP (VG): "vg0"                          |
|                       (Pool of Physical Extents, e.g. 4MB chunks)       |
+-------------------------------------------------------------------------+
                                    ^
+-------------------------------------------------------------------------+
|      PHYSICAL VOLUMES (PV): /dev/sda2, /dev/sdb1, /dev/nvme0n1p2        |
+-------------------------------------------------------------------------+

1. Physical Volumes (PV): Initialized storage disks or partitions (pvcreate /dev/sdb).
2. Volume Groups (VG): A storage pool aggregated from one or more PVs (vgcreate vg0 /dev/sdb /dev/sdc).
3. Logical Volumes (LV): Virtual partitions carved out of the VG pool (lvcreate -L 50G -n data vg0).
4. Online Resizing: With LVM, storage can be expanded on a live running production system without downtime:

   # Add 20 GB to the volume and resize the underlying ext4/xfs filesystem in one command:
   sudo lvextend -L +20G -r /dev/vg0/data
   

Mounting and /etc/fstab:

The kernel mounts filesystems at boot according to /etc/fstab. Each line specifies:

UUID=3a4f6b21-4d92-4e89-a291-0e5d1234abcd  /var  ext4  defaults,noatime,nodev  0  2

1. Device / UUID: Always use UUID= instead of /dev/sda1 to avoid drive letter reordering upon reboot.
2. Mount Point: Target directory.
3. Filesystem Type: ext4, xfs, btrfs, tmpfs.
4. Mount Options: Performance & security flags (noatime disables write updates on reads; nodev blocks device node creation; nosuid ignores SUID bits).
5. Dump Flag: Backup utility flag (0 disables).
6. FSCK Pass Order: 1 for root (/), 2 for other local drives, 0 to skip filesystem checking.

4. Advanced Permissions: SUID, SGID & Sticky Bit

Standard POSIX permissions define Read (4), Write (2), and Execute (1) for User, Group, and Others. Multi-tenant and secure production systems rely on the three Special Permission Bits:

5. User Authentication, PAM & Environment Initialization

User Databases:

• /etc/passwd: World-readable database of user accounts (username:x:UID:GID:GECOS:home:shell). The x indicates that encrypted password hashes reside securely in /etc/shadow.
• /etc/shadow: Root-readable only (0640 or 0600). Stores salted cryptographic password hashes (e.g., $6$ for SHA-512, $y$ for yescrypt) and password expiration policies.
• /etc/group: Group memberships and supplemental GIDs.

PAM (Pluggable Authentication Modules):

Applications like login, sshd, and sudo do not check passwords directly. They delegate authentication to PAM via configuration files in /etc/pam.d/:

• auth: Verifies user identity (password prompt, 2FA hardware token).
• account: Checks if the account is allowed to log in (password expired? time restrictions?).
• password: Enforces password update policies (minimum length, complexity).
• session: Prepares user environment (mounting home directory, setting resource limits via pam_limits.so).

Shell Initialization Sequence:

When a user logs in, bash executes configuration scripts in a strict deterministic order:

1. Login Shell (SSH login, console login):
   • /etc/profile (system-wide environment)
   • First matching user file: ~/.bash_profile → ~/.bash_login → ~/.profile
2. Interactive Non-Login Shell (opening a terminal window inside a desktop, tmux window, subshell):
   • /etc/bash.bashrc (system-wide)
   • ~/.bashrc (user aliases, functions, prompts)

6. systemd Architecture & Service Hardening

On modern Linux distributions, systemd is PID 1:

• It handles initial userspace boot orchestration and parallel service startup.
• It reaps orphaned zombie processes.
• It supervises background daemons using cgroups.
• It enables Socket Activation: systemd binds network listening sockets before the application starts, handing the open file descriptor to the service upon the first incoming TCP connection.

Declarative Security Sandboxing:

Modern systemd units can enforce kernel isolation directly without external container engines:

[Unit]
Description=Production Payment Microservice
After=network.target

[Service]
Type=notify
ExecStart=/opt/payment-service/bin/server
Restart=always
RestartSec=5s

# Security Sandboxing Primitives
User=payment-svc
Group=payment-svc
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# cgroups v2 Resource Governance
MemoryMax=512M
MemoryHigh=400M
CPUQuota=150%
TasksMax=100

[Install]
WantedBy=multi-user.target

7. Hands-On Practical Labs

Lab 1.1: High-Performance Log Aggregation Pipeline in Bash

Write a POSIX-compliant shell script that parses a multi-gigabyte Nginx access log, filters for HTTP 500 error codes, aggregates the count per client IP, and outputs the top 10 offending clients.

Script Source: top_error_ips.sh

#!/bin/sh
set -eu

LOG_FILE="${1:-/var/log/nginx/access.log}"

if [ ! -r "$LOG_FILE" ]; then
    echo "Error: Cannot read log file $LOG_FILE" >&2
    exit 1
fi

echo "=== Top 10 Offending IPs Triggering HTTP 5xx Errors ==="
awk '$9 ~ /^5[0-9]{2}$/ { print $1 }' "$LOG_FILE" | \
    sort | \
    uniq -c | \
    sort -rn | \
    head -n 10 | \
    awk '{ printf "%-5d requests | Client IP: %s\n", $1, $2 }'

Lab 1.2: Hardened Collaboration Directory with SGID & Sticky Bit

Configure a multi-user directory where members of the devops team collaborate:

1. Files created inherit group devops.
2. Users cannot delete files created by team members.

# 1. Create directory and group
sudo groupadd -f devops
sudo mkdir -p /srv/devops_collab

# 2. Assign ownership
sudo chown root:devops /srv/devops_collab

# 3. Apply SGID (2000) and Sticky Bit (1000) with standard 0770 permissions
sudo chmod 3770 /srv/devops_collab

# 4. Verify permission string
ls -ld /srv/devops_collab
# Expected output: drwxrws--t 2 root devops 4096 /srv/devops_collab

Lab 1.3: Inspecting Device Major/Minor Numbers and Writing a udev Rule

In this lab, we inspect kernel device properties using udevadm and write a custom rule to assign persistent symlinks and permissions.

# 1. Inspect properties of disk device /dev/sda or loop0
udevadm info --query=all --name=/dev/loop0 2>/dev/null || udevadm info --query=all --name=/dev/sda

# 2. Monitor kernel uevents live (plug in a USB drive or create a loop device to observe events)
# udevadm monitor --kernel --udev

# 3. Example udev rule to give non-root developers read/write access to a custom serial device:
# Create /etc/udev/rules.d/99-custom-serial.rules:
# SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", MODE="0666", SYMLINK+="custom_serial"

# 4. Reload and trigger udev rules
sudo udevadm control --reload-rules
sudo udevadm trigger

Lab 1.4: Dynamic LVM Storage Creation and Online Expansion

In this lab, we build a complete 3-tier LVM storage stack using two ephemeral loopback disk images, format it as ext4, and expand it online while mounted.

# 1. Create two 500MB raw disk image files
dd if=/dev/zero of=/tmp/disk1.img bs=1M count=500 status=none
dd if=/dev/zero of=/tmp/disk2.img bs=1M count=500 status=none

# 2. Attach disk images as loop devices
LOOP1=$(sudo losetup -f --show /tmp/disk1.img)
LOOP2=$(sudo losetup -f --show /tmp/disk2.img)
echo "Attached loop devices: $LOOP1 and $LOOP2"

# 3. Initialize Physical Volumes (PV)
sudo pvcreate $LOOP1 $LOOP2

# 4. Create Volume Group (VG) combining both PVs (Total ~1000MB)
sudo vgcreate lab_vg $LOOP1 $LOOP2

# 5. Create a 400MB Logical Volume (LV)
sudo lvcreate -L 400M -n lab_data lab_vg

# 6. Format as ext4 and mount
sudo mkfs.ext4 /dev/lab_vg/lab_data
sudo mkdir -p /mnt/lvm_lab
sudo mount /dev/lab_vg/lab_data /mnt/lvm_lab
df -h /mnt/lvm_lab

# 7. Perform ONLINE Expansion (Grow to 700MB without unmounting!)
sudo lvextend -L 700M -r /dev/lab_vg/lab_data
df -h /mnt/lvm_lab

# 8. Clean up
sudo umount /mnt/lvm_lab
sudo lvremove -y /dev/lab_vg/lab_data
sudo vgremove -y lab_vg
sudo pvremove -y $LOOP1 $LOOP2
sudo losetup -d $LOOP1 $LOOP2
rm -f /tmp/disk1.img /tmp/disk2.img /mnt/lvm_lab
echo "[+] LVM lab completed and cleaned up successfully!"

Lab 1.5: Deploying and Testing a Sandboxed systemd Service

In this lab, we build, install, and test a daemon constrained by systemd cgroup limits.

1. Create a Test Daemon: worker.py

#!/usr/bin/env python3
import time
import os

print(f"[*] Worker daemon started. PID: {os.getpid()}")
while True:
    time.sleep(1)

2. Create the Unit File: /etc/systemd/system/sandboxed-worker.service

[Unit]
Description=Sandboxed Python Worker
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/worker.py
Restart=always
PrivateTmp=true
ProtectSystem=strict
MemoryMax=100M
CPUQuota=25%

[Install]
WantedBy=multi-user.target

3. Enable, Start & Verify

# Copy python worker
sudo cp worker.py /opt/worker.py

# Reload systemd daemon
sudo systemctl daemon-reload

# Start and enable service
sudo systemctl start sandboxed-worker.service
sudo systemctl status sandboxed-worker.service

# Inspect active cgroup limits applied by systemd
systemd-cgls
systemctl show sandboxed-worker.service --property=MemoryMax,CPUQuota

8. Summary Checklist

• FHS Standards: Understood purpose and lifecycle of /etc, /var, /proc, /sys.
• Device Nodes & udev: Mastered character vs. block devices, major/minor numbers, and systemd-udevd netlink uevents.
• LVM Architecture: Traced the 3-tier hierarchy (PV → VG → LV) and executed live filesystem online expansion.
• Special Permissions: Mastered octal masks for SUID (4000), SGID (2000), and Sticky Bit (1000).
• Authentication & PAM: Examined /etc/shadow, Pluggable Authentication Modules, and login shell sequence.
• systemd Hardening: Deployed declarative sandboxing with ProtectSystem, MemoryMax, and PrivateTmp.