Linux syscalls - execve / execveat

The exec family of function calls—anchored by the underlying execve system call—is the primary mechanism Unix-like operating systems use to run new programs.

While fork() clones an existing process, the exec family replaces the current process’s memory, stack, heap, and code segment with an entirely new executable.

What Happens During an exec Call?

When a process calls an exec function successfully:

  1. Memory Overwrite: The kernel discards the old program image (code, stack, heap, and static variables).
  2. New Executable Loaded: The new binary (or script with a #! shebang) is loaded into the process's virtual address space.
  3. Execution Reset: The instruction pointer jumps to the entry point of the new binary (e.g., its main() function).
  4. PID Preservation: The Process ID (PID), parent PID, and resource limits remain unchanged. It is the same process, now running a different program.
  5. File Descriptors: Open file descriptors remain open across the exec boundary unless they were opened with the O_CLOEXEC flag (Close-on-Exec), in which case the kernel closes them automatically.

Crucial Rule: If exec succeeds, it never returns. The code following an exec call only runs if the system call encounters an error (such as ENOENT for file not found or EACCES for permission denied).

Note: the syscall is called execve. All the other exec* variations—such as execl, execv, execlp, execvp, and execle—are user-space C library functions (wrappers provided by glibc or musl) that format their arguments and eventually call execve (or execveat) under the hood.

Decoding the Letter Suffixes

The C standard library exposes several variations of exec. The suffix letters attached to exec indicate how parameters are passed to the function:

  e  x  e  c  [l | v] [p] [e]
               |       |   |
               |       |   └── Environment: Accepts explicit envp[] array
               |       └────── Path: Searches PATH environment variable
               └────────────── List OR Vector: How arguments are passed

The Suffix Rules

  • l (List): Arguments are passed individually as a comma-separated list of C-strings. The list must end with a NULL pointer.

  • Use case: When you know the exact number of arguments at compile time.

  • v (Vector): Arguments are passed as an array (vector) of string pointers (char *const argv[]), terminated by a NULL pointer.

  • Use case: When the number of arguments is dynamic or constructed at runtime.

  • p (Path): Takes a filename instead of an absolute path, and uses the process's PATH environment variable to locate the executable binary.

  • e (Environment): Allows you to explicitly pass a custom environment array (char *const envp[]) to the new program, overriding the inherited environment.

Map of the exec Family

Function Argument Format Path Lookup? Custom Environment? Underlying API
execl() List (arg0, arg1, ..., NULL) ❌ Exact path ❌ Inherited C Library Wrapper
execv() Vector (argv[]) ❌ Exact path ❌ Inherited C Library Wrapper
execlp() List (arg0, arg1, ..., NULL) Yes (PATH) ❌ Inherited C Library Wrapper
execvp() Vector (argv[]) Yes (PATH) ❌ Inherited C Library Wrapper
execle() List (arg0, ..., NULL, envp[]) ❌ Exact path Yes (envp[]) C Library Wrapper
execvpe() Vector (argv[]) Yes (PATH) Yes (envp[]) C Library Wrapper (GNU extension)
execve() Vector (argv[]) ❌ Exact path Yes (envp[]) Actual Kernel Syscall

Code Examples

Example A: execvp (List-based Arguments with PATH Lookup)

Commonly used in shells when running user commands like ls -l /tmp:

#include <stdio.h>
#include <unistd.h>

int main() {
    char *args[] = {"ls", "-l", "/tmp", NULL};

    printf("About to replace this process with 'ls'...\n");

    // Uses PATH to find 'ls', passes vector array
    execvp(args[0], args);

    // This code ONLY executes if execvp fails
    perror("execvp failed");
    return 1;
}

Example B: execve (The Low-Level System Call)

Requires absolute/relative paths and explicit environment definition:

#include <stdio.h>
#include <unistd.h>

int main() {
    char *path = "/bin/env";
    char *args[] = {"env", NULL};
    char *env[] = {"USER=custom_user", "MY_VAR=123", NULL};

    // Replaces process with /bin/env using custom environment variables
    execve(path, args, env);

    perror("execve failed");
    return 1;
}

Modern Extension: fexecve

POSIX.1-2008 introduced fexecve():

int fexecve(int fd, char *const argv[], char *const envp[]);

Instead of accepting a pathname string, fexecve accepts an open file descriptor (fd) pointing to an executable file (often opened using openat with O_PATH or O_RDONLY).

Why fexecve matters:

  1. Immunity to TOCTOU Race Conditions: Opening the file first and passing its descriptor guarantees that the exact binary verified/opened by the caller is executed, preventing symlink substitution or path swapping attacks during invocation.
  2. Execution from Sandboxed Contexts: Allows executing binaries pinned via directory descriptors or anonymous descriptors (memfd_create) in restricted or sandboxed runtime environments.

A chart to map this out:

User Application
  ├── execl()   ──────┐
  ├── execv()   ──────┤
  ├── execlp()  ──────┼──> C Library (glibc) ──> sys_execve() ────┐
  ├── execvp()  ──────┤   (Resolves PATH,        (Kernel)         │
  ├── execle()  ──────┘    builds argv/envp)                      ├──> Replaces
  │                                                               │    Process Image
  └── fexecve() ─────────> C Library (glibc) ──> sys_execveat() ──┘
                          (Uses /proc/self/fd   (Kernel)
                           or AT_EMPTY_PATH)

Historical Context: Why Is It Called execve Instead of exec?

In early 1970s Unix (First Edition), the original system call was actually named exec(). However, its design had limitations:

  • Early Unix exec(): Only took a path and an argument list. It did not support environment variable propagation.
  • Unix Seventh Edition (7th Edition, 1979): The Unix environment mechanism (environ) was formalized. System calls were updated to accept environment arrays explicitly, giving birth to execve (Execute with Vector arguments and Environment).

As the C standard library evolved, functions like execl, execv, and execvp were introduced as user-space wrappers to make invoking execve more convenient for developers, but the kernel interface settled permanently on execve (and later, execveat).

Loading...