System Calls & Interrupt Handling

Operating systems rely on hardware interrupts and system calls to transition execution safely across hardware protection boundaries. Interrupts allow physical hardware to notify the CPU of external asynchronous events, while system calls provide controlled software gateways for unprivileged user applications (Ring 3) to request privileged services from the kernel (Ring 0).

+-------------------------------------------------------------+
|                User Space (Ring 3 / EL0)                    |
|   Application Code -> libc wrapper -> SYSCALL / SVC         |
+-------------------------------------------------------------+
                               |
                   Hardware Privilege Gateway
                (MSR IA32_LSTAR / Vector Table)
                               |
                               v
+-------------------------------------------------------------+
|                Kernel Space (Ring 0 / EL1)                  |
|   Hardware Context Save -> Syscall Dispatch Table -> ISR    |
+-------------------------------------------------------------+
                               ^
                               |
+-------------------------------------------------------------+
|               Hardware Devices & Timers                     |
|   NIC Packet / NVMe Completion / Timer Tick -> APIC / MSI-X |
+-------------------------------------------------------------+

Hardware Interrupts, Traps, Faults, and Aborts

The CPU classifies execution interruptions into four architectural categories:

Hardware Interrupts (Asynchronous):
- Generated by external physical devices (network interfaces, disk controllers, keyboards, timer chips).
- Occur independently of the instruction stream; the CPU checks for pending interrupts at the end of every instruction cycle.
Traps (Synchronous):
- Intentional software-triggered exceptions (e.g. INT 3 for debugger breakpoints, INTO for overflow detection).
- The CPU resumes execution at the subsequent instruction.
Faults (Synchronous & Recoverable):
- Generated during instruction execution due to conditions the OS can often resolve (e.g. Page Fault #PF for demand paging/CoW, Segment Not Present #NP).
- The CPU saves the address of the faulting instruction itself, so execution re-attempts the exact same instruction once the kernel repairs the state.
Aborts (Severe & Unrecoverable):
- Severe hardware failure or corrupted system state (e.g. Machine Check Exception #MC, Double Fault #DF). Execution cannot safely resume.

Hardware Interrupt Architecture: APIC & MSI-X

Early PCs routed interrupts through two cascaded 8259 Programmable Interrupt Controllers (PIC), supporting only 15 legacy IRQ lines shared across devices.

Modern multi-core systems deploy the Advanced Programmable Interrupt Controller (APIC) on x86, or the Generic Interrupt Controller (GIC) on ARM:

[ Device: NVMe SSD ] ---> [ MSI-X Message ] --+
                                               |
[ Device: 100GbE NIC] ---> [ MSI-X Message ] -+--> PCI Express Bus
                                               |
[ System Timer Chip ] ---> [ Local APIC ] -----+
                                               |
                                               v
                        +------------------------------+
                        |       I/O APIC / Core        |
                        +------------------------------+
                               /       |        \
                              v        v         v
                          [Core 0]  [Core 1]  [Core 2]

Local APIC (LAPIC): Resides inside each CPU core, handling inter-processor interrupts (IPIs) and local timer ticks.
I/O APIC: Routes external device interrupt lines to individual CPU cores based on programmable redirection tables.
Message Signaled Interrupts (MSI / MSI-X): Modern PCIe devices do not use physical wire lines. Instead, they write a small in-memory 32-bit value to a reserved address on the host bridge. MSI-X supports up to 2,048 distinct interrupt vectors per device, enabling multi-queue network cards (NICs) to assign dedicated interrupt vectors directly to specific CPU cores without lock contention.

Two-Phase Interrupt Handling: Top-Half vs Bottom-Half

Handling an interrupt entirely within the immediate interrupt handler would freeze the CPU with interrupts disabled, dropping subsequent packets and introducing severe audio/real-time jitter.

Kernels partition interrupt handling into two distinct stages:

Hardware Interrupt Arrives
         |
         v
+--------------------------------------------------------------+
| Top-Half (Hardirq / ISR)                                     |
| - Runs with interrupts disabled (minimal duration: < 5 us)   |
| - Acknowledges hardware controller (EOI - End of Interrupt)  |
| - Copies hardware FIFO registers to kernel RAM buffer        |
| - Schedules bottom-half deferred work                        |
+--------------------------------------------------------------+
         |
         v (Interrupts Re-Enabled)
+--------------------------------------------------------------+
| Bottom-Half (Deferred Execution)                             |
| - Runs asynchronously with interrupts enabled                |
| - Performs intensive processing (TCP/IP checksums, parsing)  |
| - Linux: softirqs, tasklets, workqueues, threaded IRQs       |
| - Windows: Deferred Procedure Calls (DPCs)                   |
+--------------------------------------------------------------+

Bottom-Half Implementations Across OSes

Linux:
- softirq: 10 statically allocated, performance-critical subsystems (networking RX/TX, block I/O, timer ticks, RCU). Can execute concurrently on multiple CPUs.
- tasklet: Dynamically allocated bottom-halves built on softirqs; guaranteed never to run concurrently on multiple cores with the same instance.
- Threaded IRQs: Modern Linux paradigm converting interrupt handling into dedicated schedulable kernel threads ([irq/24-nvme]), allowing interrupt processing to be prioritized, throttled, and pre-empted by real-time schedulers.
- workqueue: Deferred execution running in normal kernel thread context; can sleep, allocate memory, and block.
Windows NT:
- Interrupt Service Routine (ISR): Executes at elevated Interrupt Request Level (DIRQL).
- Deferred Procedure Call (DPC): Queued by the ISR to run at DISPATCH_LEVEL. DPCs drain in batches before the CPU drops back to user-mode thread execution.

The Anatomy of a Modern System Call

A system call transfers execution from user space (Ring 3 / EL0) to kernel space (Ring 0 / EL1) in a tightly orchestrated hardware sequence.

Evolution of the System Call Instruction

Legacy x86 (int 0x80): Software interrupt traversing the Interrupt Descriptor Table (IDT). Required reading memory tables, checking privilege segments, and pushing 5 registers onto the stack (~300–400 CPU cycles).
Modern Fast Syscalls (SYSCALL / SYSRET on x86-64; SVC on ARM64): Bypasses the IDT completely using dedicated Model Specific Registers (MSRs) pre-programmed at kernel boot time:
- IA32_LSTAR (Long System Target Address): Holds the target 64-bit kernel entry point function pointer (entry_SYSCALL_64).
- IA32_STAR: Holds target code and stack segment selectors for Ring 0 and Ring 3.
- IA32_FMASK: Mask for RFLAGS register (automatically disables interrupts on entry).

Hardware & Software Execution Trace

1. User Application calls read(fd, buf, count)
   - libc loads arguments into registers:
     RAX = 0 (sys_read syscall number)
     RDI = fd, RSI = buf, RDX = count
2. libc executes SYSCALL instruction:
   - Hardware saves RIP into RCX
   - Hardware saves RFLAGS into R11
   - Hardware loads RIP from IA32_LSTAR (entry_SYSCALL_64)
   - Hardware switches privilege bits from Ring 3 to Ring 0
3. Kernel Entry Stub (entry_SYSCALL_64):
   - SWAPGS: Switches GS register from user TLS to kernel per-CPU data
   - Switches user RSP to kernel stack (TSS.RSP0)
   - Pushes user registers to construct struct pt_regs
4. Dispatch:
   - Bounds-checks syscall number in RAX against sys_call_table
   - Invokes sys_read(fd, buf, count)
5. Return:
   - Places return value (or -errno) in RAX
   - Restores registers from pt_regs
   - SWAPGS: Restores user GS register
   - SYSRETQ instruction: Restores RIP from RCX, RFLAGS from R11, Ring 3 mode

ABI Calling Conventions: System V vs Windows x64

Parameter	Linux / BSD / macOS (System V AMD64)	Windows NT x64
Syscall Number	`RAX`	`EAX`
Argument 1	`RDI`	`RCX`
Argument 2	`RSI`	`RDX`
Argument 3	`RDX`	`R8`
Argument 4	`R10` (diverges from `RCX` due to `SYSCALL` clobber)	`R9`
Argument 5	`R8`	Stack (`[RSP + 40]`)
Argument 6	`R9`	Stack (`[RSP + 48]`)
Shadow Space	No	32 bytes reserved home space
Error Return	`-errno` in `RAX` (negative value `-1` to `-4095`)	`NTSTATUS` (32-bit status code)

Zero-Overhead Syscall Accelerations

Entering and leaving the kernel costs ~100–300 nanoseconds, compounded significantly by KPTI (Kernel Page Table Isolation) page-table swapping. High-performance software circumvents this overhead through two architectural innovations:

1. vDSO (Virtual Dynamic Shared Object)

For read-only queries that only inspect kernel state (such as clock_gettime(), gettimeofday(), getcpu()), switching to Ring 0 is unnecessary:

The kernel maps a small page of physical RAM containing atomic timekeeping timestamps (updated every timer tick) as read-only memory into every process's virtual address space.
A small shared object (linux-vdso.so.1) is injected by the ELF dynamic linker.
When an application calls clock_gettime(CLOCK_REALTIME), libc directs the call to the vDSO implementation, which reads the current timestamp and adds CPU cycle counter offsets (RDTSC) entirely in user mode.
Execution latency drops from ~150 ns to ~10 ns with zero privilege transitions.

2. Batched Ring Buffers: Linux `io_uring`

Traditional synchronous I/O forces two system calls per transaction (write() + read()), causing CPU bottlenecks at multi-million IOPS scales.

io_uring eliminates syscall invocation per I/O operation:

Uses two shared-memory circular ring buffers allocated via mmap():
1. Submission Queue (SQ): User space writes I/O requests into the ring buffer.
2. Completion Queue (CQ): Kernel places completion results into the ring buffer.
With kernel polling mode (IORING_SETUP_SQPOLL), a dedicated kernel thread continuously polls the Submission Queue. Applications submit and harvest thousands of asynchronous I/O requests with zero system call transitions.

Security Filtering: `seccomp-bpf`

Unrestricted access to the entire kernel system call table (~450 syscalls in Linux) creates an enormous attack surface.

Seccomp-BPF (Secure Computing with Berkeley Packet Filters) allows applications (such as Docker, Chrome sandboxes, and systemd services) to attach an in-kernel BPF filter to the syscall entry dispatcher:

Evaluates the syscall number and memory arguments before the kernel executes the routine.
Actions: Allow, Kill (SIGSYS), Log, or return a fake errno (e.g. EPERM).
Blocks unused capabilities (e.g. preventing a web browser render process from invoking ptrace, mount, or reboot).