Linux - send(), recv(), sendto(), recvfrom(), sendmsg(), recvmsg()

Once a socket is connected or bound, data transmission occurs through the socket I/O system call family. While standard POSIX read() and write() work on sockets, the dedicated socket I/O system calls provide essential flags for non-blocking operations, signal suppression, datagram peer routing, ancillary metadata control, and packet batching:

Linux Socket I/O Syscall Hierarchy:

          Basic Stream I/O                   Connectionless Datagrams
        (Peer already known)                  (Per-packet addressing)
         ┌────────────────┐                     ┌────────────────┐
         │ send()  recv() │                     │sendto/recvfrom │
         └───────┬────────┘                     └───────┬────────┘
                 │                                      │
                 ▼                                      ▼
       ┌────────────────────────────────────────────────────────┐
       │ sendmsg() / recvmsg()                                  │
       │ Vectored I/O (iovec) + Ancillary Control Data (cmsg)   │
       │ (Passing File Descriptors SCM_RIGHTS, UIDs, Timestamps)│
       └──────────────────────────┬─────────────────────────────┘
                                  │
                                  ▼
       ┌────────────────────────────────────────────────────────┐
       │ sendmmsg() / recvmmsg()                                │
       │ Multi-message batching (1 syscall = hundreds of pkts)  │
       └────────────────────────────────────────────────────────┘

1. send() and recv(): Stream I/O with Control Flags

#include <sys/socket.h>

ssize_t send(int sockfd, const void *buf, size_t len, int flags);
ssize_t recv(int sockfd, void *buf, size_t len, int flags);

On a connected socket, send(fd, buf, len, 0) is equivalent to write(fd, buf, len), and recv(fd, buf, len, 0) is equivalent to read(fd, buf, len). The critical difference is the flags bitmask:

Essential send() Flags

• MSG_NOSIGNAL: Suppresses the fatal SIGPIPE signal.
  • When writing to a socket whose remote peer has already closed or reset the connection, the Linux kernel default is to emit a SIGPIPE signal to your process.
  • If your process does not install a signal handler or ignore SIGPIPE, the kernel terminates your entire application immediately!
  • Passing MSG_NOSIGNAL suppresses the signal and instead returns -1 with errno == EPIPE, allowing safe, graceful error recovery in application logic.
• MSG_DONTWAIT: Enables non-blocking operation for this single call only, without needing to toggle the socket's persistent O_NONBLOCK state via fcntl(). If no buffer space is available, it returns -1 with errno == EAGAIN (or EWOULDBLOCK).
• MSG_MORE: Tells the kernel that more data is immediately following. The TCP stack will withhold transmitting a partial segment (similar to setting TCP_CORK), buffering the data until a full MTU frame is assembled or until a final send() without MSG_MORE is executed.

Essential recv() Flags

• MSG_PEEK: "Peeks" at incoming data without consuming it. The bytes are copied into your user buffer, but remain queued in the socket receive buffer for subsequent recv() or read() calls.
• MSG_WAITALL: Blocks until the entire requested len of bytes is received. (The call may still return fewer bytes if a signal interrupts it, the connection closes, or an error occurs).
• MSG_DONTWAIT: Returns immediately with EAGAIN if the kernel socket receive queue is empty.
• MSG_TRUNC: For raw or packet sockets, returns the real packet length even if it exceeded the user buffer size.

The Zero-Return Rule in recv()

When recv() returns 0, it indicates that the remote peer has performed an orderly shutdown (sent a TCP FIN packet). The stream has reached End-Of-File (EOF). The application must close the descriptor or cleanly shut down its end.

2. sendto() and recvfrom(): Connectionless Datagrams

#include <sys/socket.h>

ssize_t sendto(int sockfd, const void *buf, size_t len, int flags,
               const struct sockaddr *dest_addr, socklen_t addrlen);

ssize_t recvfrom(int sockfd, void *buf, size_t len, int flags,
                 struct sockaddr *src_addr, socklen_t *addrlen);

For connectionless protocols like UDP (SOCK_DGRAM) and raw sockets (SOCK_RAW), sockets do not have an established peer. Every packet must declare its destination and inspect its sender:

• sendto(): Directs each individual datagram to dest_addr.
• recvfrom(): Fills src_addr with the IP address and port of the sender that transmitted the datagram.

// UDP Server Echo Loop
char buffer[1500];
struct sockaddr_storage client_addr;
socklen_t addr_len = sizeof(client_addr);

// 1. Receive packet and capture who sent it
ssize_t bytes_in = recvfrom(udp_fd, buffer, sizeof(buffer), 0,
                            (struct sockaddr *)&client_addr, &addr_len);

// 2. Reply directly to the sender address
ssize_t bytes_out = sendto(udp_fd, buffer, bytes_in, 0,
                           (struct sockaddr *)&client_addr, addr_len);

3. sendmsg() and recvmsg(): The Advanced Control Hub

#include <sys/socket.h>

ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags);
ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);

sendmsg() and recvmsg() are the most capable I/O primitives in POSIX networking. They combine two superpowers into a single system call:

1. Vectored (Scatter-Gather) I/O: Sending from or reading into multiple non-contiguous memory buffers via struct iovec (identical to readv & writev).
2. Ancillary Control Data: Passing out-of-band kernel metadata and operating system capabilities via msg_control.

Anatomy of struct msghdr

struct msghdr {
    void         *msg_name;       /* Optional address (for UDP / raw sockets) */
    socklen_t     msg_namelen;    /* Size of address */
    struct iovec *msg_iov;        /* Scatter-gather vector array */
    size_t        msg_iovlen;     /* Number of elements in msg_iov */
    void         *msg_control;    /* Ancillary data buffer (cmsghdr) */
    size_t        msg_controllen; /* Ancillary data buffer length */
    int           msg_flags;      /* Flags on received message */
};

Passing File Descriptors over Unix Sockets (SCM_RIGHTS)

A fundamental pattern in Linux system architecture (used by Docker, systemd, Wayland, and Chromium) is passing an open file descriptor from one process to an entirely separate process over a Unix Domain Socket:

Passing a File Descriptor via SCM_RIGHTS:

   Process A (Privileged / Root)               Process B (Worker Sandbox)
   ─────────────────────────────               ──────────────────────────
   1. Opens /etc/secret.key (fd 4)                         │
   2. Packages fd 4 into struct msghdr                     │
      as SCM_RIGHTS ancillary data                         │
   3. Calls sendmsg(unix_sock, &msg, 0)                    │
                  │                                        │
                  └────────► [ Linux Kernel ] ─────────────┘
                             Increments struct file refcount (get_file)
                             Both FDs point to the SAME struct file!
                             Allocates unused slot in Process B (e.g., fd 9)
                                                           │
                                                           ▼
                                           Calls recvmsg(unix_sock, &msg, 0)
                                           Directly reads /etc/secret.key via fd 9!

[!NOTE] Does SCM_RIGHTS clone struct file? No. The kernel does not clone struct file. Both processes' file descriptors point to the exact same struct file (Open File Description) in kernel memory:

• Shared Offset (f_pos): If Process A reads 100 bytes from a regular file, the offset advances for Process B as well.
• Shared Status Flags (f_flags): Toggling flags like O_NONBLOCK via fcntl(F_SETFL) impacts both processes.
• Independent Descriptor Flags: Flags like FD_CLOEXEC are private to each process's file table (files_struct). Use the MSG_CMSG_CLOEXEC flag with recvmsg() to enable FD_CLOEXEC atomically on the receiver side.

To encode or decode ancillary control data, the kernel provides helper macros (CMSG_FIRSTHDR, CMSG_NXTHDR, CMSG_DATA, CMSG_SPACE):

// Packing a file descriptor into an SCM_RIGHTS control message
char dummy_byte = 'X';
struct iovec iov = { .iov_base = &dummy_byte, .iov_len = 1 };

char cmsg_buf[CMSG_SPACE(sizeof(int))];
struct msghdr msg = {
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_control = cmsg_buf,
    .msg_controllen = sizeof(cmsg_buf),
};

struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS;
cmsg->cmsg_len = CMSG_LEN(sizeof(int));

// Inject the file descriptor to pass
int fd_to_share = open("/etc/shadow", O_RDONLY);
*((int *)CMSG_DATA(cmsg)) = fd_to_share;

sendmsg(unix_socket_fd, &msg, 0);

Read more about control messages in Linux - SCM Credentials.

4. sendmmsg() and recvmmsg(): Multi-Message Batching

#define _GNU_SOURCE
#include <sys/socket.h>

int sendmmsg(int sockfd, struct mmsghdr *msgvec, unsigned int vlen, int flags);
int recvmmsg(int sockfd, struct mmsghdr *msgvec, unsigned int vlen, int flags,
             struct timespec *timeout);

In high-throughput networking (such as 10GbE/100GbE DNS servers, QUIC engines, or real-time gaming clusters), packet transmission rate easily reaches 500,000 packets per second.

Executing one system call per packet means 500,000 user-to-kernel context switches every second, burning up to 40% of the CPU purely on Meltdown/Spectre mitigations and register saving.

Linux introduced sendmmsg() and recvmmsg() to transmit or receive arrays of messages in a single atomic syscall:

struct mmsghdr {
    struct msghdr msg_hdr;  /* Actual message structure */
    unsigned int  msg_len;  /* Number of bytes transmitted/received */
};

High-Throughput Packet Batching Example

#define BATCH_SIZE 32
struct mmsghdr msgs[BATCH_SIZE];
struct iovec iovs[BATCH_SIZE];
char buffers[BATCH_SIZE][1500];

for (int i = 0; i < BATCH_SIZE; i++) {
    iovs[i].iov_base = buffers[i];
    iovs[i].iov_len = sizeof(buffers[i]);
    msgs[i].msg_hdr.msg_iov = &iovs[i];
    msgs[i].msg_hdr.msg_iovlen = 1;
}

// Read up to 32 complete datagrams in a single context switch!
int pkts_read = recvmmsg(udp_fd, msgs, BATCH_SIZE, MSG_WAITFORONE, NULL);
for (int i = 0; i < pkts_read; i++) {
    printf("Packet %d: received %u bytes\n", i, msgs[i].msg_len);
}

The MSG_WAITFORONE flag blocks until at least one packet arrives, then reaps as many additional packets as are already available in the socket buffer without blocking further, returning immediately to user space.

Comparison & Decision Matrix

System Call	Best Suited For	Key Capabilities	Overhead Level
read() / write()	Generic file-descriptor abstractions	Universal interface, works on pipes, files, and sockets	Low
send() / recv()	Connected TCP / Unix streams	MSG_NOSIGNAL (suppress crash), MSG_PEEK, MSG_DONTWAIT	Low
sendto() / recvfrom()	Connectionless UDP / Raw frames	Captures sender and designates destination per packet	Low
sendmsg() / recvmsg()	Complex protocols & IPC	Vectored I/O (iovec), passing FDs (SCM_RIGHTS), timestamps	Medium
sendmmsg() / recvmmsg()	High-packet-rate UDP, DNS, QUIC	Multi-packet vector batching, amortizes context switches	Lowest per-packet cost

Frequently Asked Questions (FAQ)

Does SCM_RIGHTS clone struct file?

No. The Linux kernel does not clone struct file. When you pass an open file descriptor over a Unix Domain Socket via SCM_RIGHTS, the kernel resolves the sender's descriptor to its underlying struct file * (Open File Description), calls get_file() to increment its reference count (f_count++), and installs a pointer to that exact same object into the receiving process's private descriptor table (files_struct).

This shared-object architecture has critical operational consequences:

• Shared File Cursor (f_pos): If Process A reads 100 bytes from a regular file, the offset advances for Process B as well.
• Shared Status Flags (f_flags): Modifying flags like O_NONBLOCK via fcntl(F_SETFL) affects both processes immediately.
• Independent Descriptor Flags: Flags like FD_CLOEXEC are private to each process's file table (files_struct). To set it for the receiver, pass MSG_CMSG_CLOEXEC to recvmsg().
• Independent Lifetime: Closing the descriptor in Process A only decrements f_count; the underlying resource remains open until Process B also closes its descriptor.

Why must you send at least 1 byte of payload data when passing a file descriptor?

A sendmsg() call passing an SCM_RIGHTS control message without any normal payload (msg_iovlen = 0 or 0 bytes) will cause the receiving recvmsg() call to return 0. In POSIX socket semantics, a return value of 0 denotes End-Of-File (EOF), which indicates that the peer has closed the connection. Passing at least 1 dummy byte ensures recvmsg() returns a positive byte count, confirming successful message and descriptor delivery.

Can file descriptors be passed over TCP/IP sockets?

No. SCM_RIGHTS is exclusively supported on local Unix Domain Sockets (AF_UNIX). Passing an ancillary control message containing SCM_RIGHTS over an AF_INET or AF_INET6 TCP/UDP socket will fail with errno == EOPNOTSUPP (Operation not supported).

What is the difference between send() and write() on a Linux socket?

On a connected stream socket, write(fd, buf, len) is strictly equivalent to send(fd, buf, len, 0). The fundamental difference is that send() accepts a flags bitmask:

• MSG_NOSIGNAL: Suppresses the fatal SIGPIPE signal when writing to a closed peer, returning EPIPE instead.
• MSG_DONTWAIT: Performs non-blocking transmission for that single invocation without toggling the socket's persistent O_NONBLOCK state.
• MSG_MORE: Buffers partial frames until full MTU packets can be transmitted together.

Related Topics

• Socket Lifecycle Syscalls (socket, bind, listen, accept4): Setting up endpoints and managing connection queues.
• Socket Options Syscalls (setsockopt, getsockopt): Configuring TCP_NODELAY, SO_REUSEPORT, and socket buffers.
• Linux - Zero-Copy Networking: Bypassing kernel-user data copying via sendfile(), splice(), and MSG_ZEROCOPY.
• Linux - SCM Control Messages: Deep dive into descriptor passing and identity verification.
• Linux - struct msghdr vs. struct cmsghdr: Detailed structural comparison, memory layouts, and CMSG alignment macros.