Linux syscalls - open / openat

In POSIX-compliant systems, the open family of system calls is responsible for converting pathnames into file descriptors (fd). Over decades of Unix evolution, these calls evolved from simple path lookup mechanisms into robust, directory-relative APIs designed to eliminate race conditions and simplify thread safety.

Primary Entry Points: `open()` vs `openat()`

`open()` / `open2()`

The classic Unix system call to open or create a file using an absolute or process-relative path.

int open(const char *pathname, int flags, ... /* mode_t mode */);

Behavior: Resolves pathname starting from the current working directory (CWD) if relative, or system root (/) if absolute.
Limitations: Inherently prone to path-traversal race conditions in multi-threaded programs or secure systems, because the current working directory is process-global.

`openat()`

Introduced in POSIX.1-2008 to solve CWD races and streamline directory traversal.

int openat(int dirfd, const char *pathname, int flags, ... /* mode_t mode */);

Relative Resolution: If pathname is relative, it is resolved relative to the directory referenced by dirfd rather than the process's current working directory.
Special Descriptor (AT_FDCWD): If dirfd is set to AT_FDCWD, openat() behaves identically to open(), resolving relative to the process's CWD.
Absolute Path Fallback: If pathname is an absolute path (starts with /), dirfd is ignored.

Why openat() Solves Race Conditions & Security Issues?

Thread Safety: In multi-threaded applications, changing the working directory via chdir() affects all threads in the process. Using openat() with directory descriptors allows individual threads to safely traverse separate folder hierarchies concurrently without changing global state.
Path Traversal Protection: By pinning a directory descriptor (dirfd), applications can open files within nested subdirectories relative to that pinned handle, preventing malicious actors on shared filesystems from swapping parent directories via symlink attacks during path resolution.

Complementary Open-Related POSIX System Calls

Beyond standard path opening, POSIX provides specialized calls to manage anonymous files, non-path file descriptors, and directory streams.

creat(): Equivalent to open(pathname, O_CREAT | O_WRONLY | O_TRUNC, mode). Historic shortcut from early Unix when open() flags were limited. Largely redundant today.
openat2() (Linux-specific extension): Extends openat() with a extensible struct parameter (struct open_how) and flags like RESOLVE_BNEATH or RESOLVE_NO_SYMLINKS to strictly restrict path resolution within target mount/directory trees.
memfd_create() (Linux/FreeBSD extension): Creates an anonymous, unlinked file in RAM/tmpfs and returns an open fd. Widely used for sandboxing, inter-process communication (IPC), and shared memory without filesystem artifacts.
opendir() / fdopendir(): Opens directory streams (DIR *) for reading directory entries (readdir). fdopendir() converts an existing dirfd into a DIR * stream.

Essential Flags Driving `open` / `openat` Behavior

The behavior of open calls is heavily driven by bitwise-ORed flags passed to flags:

Access Modes (Mutually Exclusive - Pick One)

O_RDONLY: Open for read-only.
O_WRONLY: Open for write-only.
O_RDWR: Open for reading and writing.

Operational & Safety Flags

O_CREAT: Create the file if it does not exist (requires the mode parameter).
O_EXCL: Used with O_CREAT. Fails if the file already exists. Ensures atomic file creation (prevents race conditions when creating lock files).
O_TRUNC: If the file exists and is opened for writing, truncate its length to 0 bytes.
O_APPEND: Forces all write operations to seek to the end of the file prior to each write (atomic append across processes).
O_CLOEXEC: Sets the Close-on-Exec flag automatically upon opening. Prevents the file descriptor from leaking into child processes during execve(). (Essential for multi-threaded thread safety).
O_NOFOLLOW: Fails the open if the tail component of pathname is a symbolic link.
O_DIRECTORY: Fails the open if pathname is not a directory. Essential when getting a directory descriptor for subsequent openat() calls.
O_NONBLOCK / O_NDELAY: Opens the file in non-blocking mode (critical for FIFOs, sockets, and character devices).

What does "open" mean?

The direct output and primary job of open() (and openat()) is to return a file descriptor (fd).

For example, when a VMM calls open("/dev/kvm"), it opens a file descriptor to the Linux KVM kernel module. This file descriptor acts as the fundamental api entry point into the CPU's hardware virtualization capabilities (Intel VT-x or AMD-V). By itself, opening /dev/kvm just checks if KVM is available and gives the VMM a handle to issue commands. Once that file descriptor is open, the VMM uses a sequence of ioctl() system calls to build a virtual machine from scratch.

However, calling open() triggers several critical kernel-side state changes and operations beyond just assigning an integer handle.

Key Secondary Effects of `open()`

While getting an fd is the end result, invoking open() allows you to request specific kernel actions:

1. File Creation & Modification

Atomic File Creation: Passing O_CREAT creates a new inode on disk if the file doesn't already exist. Combining it with O_EXCL guarantees atomic creation (failing if it already exists), which is used for creating lock files and avoiding race conditions.
Truncation: Passing O_TRUNC immediately shrinks an existing file's size to 0 bytes upon opening, discarding its contents.

2. Access Rights & Permission Validation

open() is the primary gatekeeper for filesystem permissions. The kernel checks your process's UID/GID against the file's mode bits (rwx) or ACLs. Once an fd is granted with a specific mode (e.g., O_RDONLY), subsequent reads or writes using that fd skip permission re-checks.

3. Creating an Open File Description (Kernel State)

When you call open(), the kernel creates an internal Open File Description (an struct file in Linux), which tracks state shared across operations:

File Offset / Position: The initial seek position (usually byte 0, or the end if O_APPEND is set).
Access Flags: Storing flags like O_NONBLOCK or O_APPEND that govern how future read() and write() calls behave.

4. Hardware and Driver Side Effects

Opening a file path in Unix isn't limited to regular files on disk:

Character/Block Devices (/dev/*): Calling open() on a device driver (e.g., /dev/tty, /dev/nvme0n1, /dev/gpu) triggers the driver's custom .open() hook. This can initialize hardware, allocate device buffers, or lock access to a device.
FIFOs / Named Pipes: Calling open() on a FIFO blocks the calling thread until another process opens the other end (read vs. write synchronization).

5. Obtaining Non-I/O Metadata Handles (`O_PATH`)

By passing O_PATH on Linux, openat() returns an fd that explicitly cannot be used for reading or writing file data. Its sole purpose here is to act as a lightweight, pin-point handle for metadata operations (like statx, fchdir, or fchmodat) without performing file permission checks or loading file contents.

What happens if you open a file multiple times?

When you open the same file multiple times—whether in the same process or across different processes—the Linux kernel creates a new Open File Description (struct file) for every successful call to open() or openat().

Each call returns a distinct integer file descriptor (fd), but all of these descriptors ultimately point to the same underlying inode in memory.

The Three-Tier Kernel Structure

To understand what happens, it helps to look at the three layers the kernel uses to track opened files:

Process File Descriptor Table
   [fd 3] ────────┐
   [fd 4] ──────┐ │
                │ │
System-Wide Open File Descriptions (Kernel Memory)
   [struct file A] (offset: 0,   flags: O_RDONLY) ──┐
   [struct file B] (offset: 128, flags: O_WRONLY) ──┼──> Inode Table (Disk Object)
                                                    │       [struct inode]
   [struct file C] (offset: 0,   flags: O_WRONLY) ──┘       (File metadata, disk blocks)

File Descriptor (fd): Per-process integer index (e.g., 3, 4) pointing to an open file description.
Open File Description (struct file): System-wide kernel object created on every open() call. Stores state unique to that open instance, such as:
- Current file position / offset (f_pos)
- Access mode and status flags (O_RDONLY, O_APPEND, O_NONBLOCK)
Inode (struct inode): System-wide single kernel object representing the actual file on disk (metadata, size, permissions, data blocks).

Key Behaviors of Multiple Opens

1. Independent File Offsets

Because each open() creates a separate struct file, each descriptor maintains its own read/write offset:

If Process A opens file.txt (fd 3) and reads 100 bytes, its offset advances to 100.
If Process B (or Process A calling open() a second time to get fd 4) reads from its descriptor, it starts reading at byte 0, unaffected by the other descriptor's position.

2. Simultaneous Writes and Concurrent Data Overwrites

If two descriptors open the file for writing without synchronization:

They will overwrite each other's data because both start writing at offset 0 by default.
The O_APPEND Exception: If both open calls specify O_APPEND, the kernel guarantees that every write() atomically seeks to the current end of the file (at the inode level) before writing, preventing data from being overwritten.

3. Independent Access Modes and Status Flags

Each open() call can request different permissions and operational flags:

One open() can request O_RDONLY, while another requests O_WRONLY.
Changing file flags using fcntl(fd, F_SETFL, ...) on fd 3 will not affect fd 4.

4. Shared Inode Operations

Because all descriptors point to the same underlying inode:

Modifications are immediately visible: If fd 3 writes data, fd 4 will read that newly written data immediately (subject to OS page cache consistency).
Traces and Attributes are Shared: Calling ftruncate(fd3, 0) truncates the actual file on disk, instantly affecting what fd4 sees.
File Locking (flock vs fcntl locks): BSD file locks (flock) operate on the open file description, whereas POSIX record locks (fcntl) operate on the (process, inode) pair.

How This Differs from `dup()` or `fork()`

It is important to distinguish opening a file multiple times from duplicating an existing descriptor:

Mechanism	System Calls	File Descriptors	Open File Description (`struct file`)	File Offset (`f_pos`)
Multiple Opens	`open()` + `open()`	Distinct (`3` & `4`)	Separate objects created	Independent
Descriptor Duplication	`dup(fd)` / `dup2()`	Distinct (`3` & `4`)	Shared same object	Shared (moving one moves the other)
Process Forking	`fork()`	Same `fd` in parent & child	Shared same object	Shared across processes