CHEATSHEETSTOOLSBOOKSHELFABOUT中文

Linux - setsockopt(), getsockopt(), getsockname(), getpeername()

In Linux socket programming, creating a socket with default attributes is rarely sufficient for production servers. High-performance networking requires fine-tuning kernel protocol engines, configuring port reusability across multiple worker processes, manipulating buffer sizes, and discovering local/remote connection topology:

Socket Configuration & Introspection Syscalls:

               Local Socket Configuration
               ──────────────────────────
           setsockopt()          getsockopt()
           (Tune behaviors)      (Query settings)
                  │                     │
                  ▼                     ▼
       ┌────────────────────────────────────────────────────────┐
       │ Kernel Socket Layers & Options                         │
       │ SOL_SOCKET   : SO_REUSEADDR, SO_REUSEPORT, SO_KEEPALIVE│
       │ IPPROTO_TCP  : TCP_NODELAY, TCP_CORK, TCP_QUICKACK     │
       │ IPPROTO_IP   : IP_TOS, IP_TTL, IP_PKTINFO              │
       └────────────────────────────────────────────────────────┘
                  ▲                     ▲
                  │                     │
           getsockname()         getpeername()
           (Who am I?)           (Who is the client?)
               ──────────────────────────
               Address & Port Discovery

1. System Call Signatures

#include <sys/socket.h>

int setsockopt(int sockfd, int level, int optname,
               const void *optval, socklen_t optlen);

int getsockopt(int sockfd, int level, int optname,
               void *optval, socklen_t *optlen);

int getsockname(int sockfd, struct sockaddr *addr, socklen_t *addrlen);
int getpeername(int sockfd, struct sockaddr *addr, socklen_t *addrlen);
  • level: The protocol layer that interprets the option:
    • SOL_SOCKET: Generic socket layer (independent of underlying protocol).
    • IPPROTO_TCP: TCP protocol stack specific options.
    • IPPROTO_IP / IPPROTO_IPV6: IP network layer options.
  • Return value: 0 on success, or -1 on error with errno set.

2. Essential Production Socket Options

SO_REUSEADDR (Bypassing TIME_WAIT on Restart)

When a server process terminates while client connections are open, the TCP connection enters the TIME_WAIT state (lasting 60 seconds on Linux, defined by TCP_TIMEWAIT_LEN).

If you restart the server immediately without SO_REUSEADDR, the subsequent call to bind() fails with EADDRINUSE: Address already in use.

int enable = 1;
if (setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(enable)) == -1) {
    perror("setsockopt SO_REUSEADDR");
}
  • What it does: Allows binding to a port that currently holds sockets in the TIME_WAIT state, provided the listening IP address matches.
  • Rule of thumb: Every TCP server should set SO_REUSEADDR before calling bind().

SO_REUSEPORT (Kernel Load Balancing Across Worker Processes)

Introduced in Linux 3.9, SO_REUSEPORT fundamentally transformed multi-process server architecture (used by Nginx, Envoy, and HAProxy):

Traditional Worker Model vs. SO_REUSEPORT Model:

Traditional Model (Contention Bottleneck):
                [ Incoming TCP Connections ]
                             │
                             ▼
               [ Single Master Listen Socket ]
               (All workers contend on accept() lock)
                  ┌──────────┼──────────┐
                  ▼          ▼          ▼
              Worker 1   Worker 2   Worker 3

SO_REUSEPORT Model (Kernel-Level Load Balancing):
                [ Incoming TCP Connections ]
                             │
                             ▼ (Kernel 4-tuple Hash)
                  ┌──────────┼──────────┐
                  ▼          ▼          ▼
              [Queue 1]  [Queue 2]  [Queue 3]
                  │          │          │
                  ▼          ▼          ▼
              Worker 1   Worker 2   Worker 3
              (Each has its own private listening socket fd!)
  • What it does: Allows multiple independent processes or threads to bind to the exact same IP address and port.
  • Kernel distribution: The Linux kernel distributes incoming connections across all listening sockets using a deterministic hash of the connection 4-tuple (Source IP, Source Port, Dest IP, Dest Port).
  • Benefits:
    • Completely eliminates lock contention on a shared accept queue.
    • Eliminates "Thundering Herd" wakeups.
    • Zero-downtime rolling upgrades: spin up new worker processes on the same port, then shut down the old workers.
int enable = 1;
setsockopt(listen_fd, SOL_SOCKET, SO_REUSEPORT, &enable, sizeof(enable));

TCP_NODELAY (Disabling Nagle's Algorithm)

By default, the TCP stack enables Nagle's Algorithm, which coalesces small outgoing packets until an entire MTU is filled or until an acknowledgment (ACK) is received for previously sent data.

While efficient for batch transfers, Nagle's algorithm is catastrophic for low-latency request-response protocols (like HTTP/2, gRPC, Redis, and WebSocket games), where it interacts with the remote peer's TCP Delayed ACK timer to cause artificial 40ms to 200ms latency spikes.

#include <netinet/tcp.h>

int enable = 1;
// Disable Nagle's algorithm: send small packets immediately!
setsockopt(conn_fd, IPPROTO_TCP, TCP_NODELAY, &enable, sizeof(enable));

TCP_CORK (Opposite of TCP_NODELAY)

TCP_CORK forces the kernel to withhold sending partial frames until the buffer is "uncorked" or a full MTU is gathered. It is frequently used when sending a custom HTTP header followed by a large file via sendfile(), ensuring the header and payload merge into minimal packets.

SO_KEEPALIVE (Detecting Dead Peers Across Silent Networks)

TCP is designed to be completely silent when no data is being sent. If a client unplugs its network cable or experiences a power outage, the server can hold the open socket in kernel memory indefinitely, leaking file descriptors.

Enabling SO_KEEPALIVE tells the kernel to send periodic probe packets:

int enable = 1;
setsockopt(conn_fd, SOL_SOCKET, SO_KEEPALIVE, &enable, sizeof(enable));

// Optional: Tune keepalive parameters per socket (Linux specific)
int idle = 60;     // Start probing after 60 seconds of inactivity
int interval = 10; // Send probe every 10 seconds
int count = 5;     // Drop connection after 5 failed probes

setsockopt(conn_fd, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle));
setsockopt(conn_fd, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval));
setsockopt(conn_fd, IPPROTO_TCP, TCP_KEEPCNT, &count, sizeof(count));

SO_RCVBUF and SO_SNDBUF (Tuning Buffer Sizes)

Sets the maximum socket receive and transmit buffer sizes in bytes:

int bufsize = 1024 * 1024; // 1 MB
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &bufsize, sizeof(bufsize));
  • Linux Note: The kernel doubles the value passed in setsockopt() to reserve internal memory for struct sk_buff book-keeping overhead.
  • Auto-Tuning: Modern Linux kernels auto-tune TCP buffer sizes automatically (via /proc/sys/net/ipv4/tcp_rmem and tcp_wmem). Manually overriding SO_RCVBUF disables the kernel's dynamic bandwidth-delay product (BDP) auto-tuning. Prefer system-wide sysctl tuning unless handling UDP datagram bursts.

SO_LINGER (Controlling close() Behavior)

Controls how close() behaves when pending unsent data remains in the socket transmit queue:

struct linger sl = {
    .l_onoff = 1,   // Non-zero = enable linger
    .l_linger = 5,  // Timeout in seconds
};
setsockopt(conn_fd, SOL_SOCKET, SO_LINGER, &sl, sizeof(sl));
  • If l_onoff == 0: close() returns immediately, and the kernel attempts to flush unsent data in the background.
  • If l_onoff != 0 and l_linger == 0: Calling close() aborts the connection immediately, discarding unsent data and sending an RST to the peer (skipping the normal FIN handshake and bypassing TIME_WAIT).

3. Address Introspection: getsockname() and getpeername()

Once a socket is initialized, how do you find out who is on either end of the wire?

struct sockaddr_storage addr;
socklen_t addr_len = sizeof(addr);

// 1. Discover local identity (e.g. which ephemeral port was assigned)
if (getsockname(fd, (struct sockaddr *)&addr, &addr_len) == 0) {
    if (addr.ss_family == AF_INET) {
        struct sockaddr_in *s = (struct sockaddr_in *)&addr;
        char ip[INET_ADDRSTRLEN];
        inet_ntop(AF_INET, &s->sin_addr, ip, sizeof(ip));
        printf("Local address: %s:%d\n", ip, ntohs(s->sin_port));
    }
}

// 2. Discover remote peer identity (who connected to us)
if (getpeername(fd, (struct sockaddr *)&addr, &addr_len) == 0) {
    if (addr.ss_family == AF_INET) {
        struct sockaddr_in *s = (struct sockaddr_in *)&addr;
        char ip[INET_ADDRSTRLEN];
        inet_ntop(AF_INET, &s->sin_addr, ip, sizeof(ip));
        printf("Remote peer: %s:%d\n", ip, ntohs(s->sin_port));
    }
}
  • getsockname(): Essential when calling bind(fd, ..., port = 0) to find out which random port the OS assigned to your service.
  • getpeername(): Used by proxies, authentication engines, and access logging to record the client IP of an accepted connection.

Quick Reference Summary

Option Name Level Type Purpose
SO_REUSEADDR SOL_SOCKET int (bool) Allows binding to port in TIME_WAIT state.
SO_REUSEPORT SOL_SOCKET int (bool) Kernel distributes incoming connections across worker sockets.
SO_KEEPALIVE SOL_SOCKET int (bool) Enables periodic TCP keepalive probes.
SO_RCVBUF / SO_SNDBUF SOL_SOCKET int (bytes) Sets socket receive/send buffer capacity.
SO_LINGER SOL_SOCKET struct linger Controls graceful flush vs RST abort on close().
SO_ERROR SOL_SOCKET int (error) Reads and clears pending asynchronous socket error (non-blocking connect).
TCP_NODELAY IPPROTO_TCP int (bool) Disables Nagle's algorithm for low latency.
TCP_CORK IPPROTO_TCP int (bool) Delays transmission to coalesce packets into full MTUs.
getsockname() Syscall sockaddr * Retrieves local IP and port bound to socket.
getpeername() Syscall sockaddr * Retrieves remote connected peer IP and port.

Frequently Asked Questions (FAQ)

What is the difference between SO_REUSEADDR and SO_REUSEPORT?

SO_REUSEADDR allows a server to bind to a port currently in the TIME_WAIT state, while SO_REUSEPORT allows multiple independent processes or threads to bind to the exact same port for kernel-level connection load balancing. SO_REUSEADDR is essential for rapid server restarts without waiting 60–120 seconds for old connections to drain from TIME_WAIT. SO_REUSEPORT (introduced in Linux 3.9) distributes incoming client connections across multiple worker sockets using a 4-tuple hash, eliminating user-space accept-mutex locks and the "thundering herd" problem.

Why does the Linux kernel double the buffer size passed to SO_RCVBUF and SO_SNDBUF?

The kernel doubles the requested byte count to reserve overhead space for internal kernel bookkeeping data structures (struct sk_buff). When an application executes setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, sizeof(val)), the kernel allocates val * 2 bytes (capped by /proc/sys/net/core/rmem_max). Half of this memory is reserved for the raw application payload data, while the other half accounts for packet headers, TCP control blocks, and socket buffer metadata.

When should you enable TCP_NODELAY?

Enable TCP_NODELAY whenever an interactive or RPC application sends small, latency-sensitive request-response messages. TCP_NODELAY disables Nagle's algorithm, which otherwise buffers small outgoing chunks until a full MSS (Maximum Segment Size) packet accumulates or an outstanding ACK is received. When paired with the receiver's TCP Delayed ACK (which delays ACKs by up to 40ms–200ms), Nagle's algorithm creates artificial latency spikes. Disabling it ensures every send() packet is transmitted across the wire immediately.

How do you determine the ephemeral port assigned to a socket?

Call getsockname() after calling bind() with port 0 or after connect() has initiated. When you pass port number 0 to bind(), the kernel selects an available ephemeral port from the range defined in /proc/sys/net/ipv4/ip_local_port_range. Calling getsockname(fd, (struct sockaddr *)&addr, &len) populates addr.sin_port with the network-byte-order port number chosen by the kernel.

Related Topics

Recommended ReadingSponsored
The Linux Programming Interface
Essential Linux Bible
The Linux Programming Interface
By Michael KerriskView on Amazon
Core & Architecture
Process & Scheduling
Memory & Executables
IPC & Networking
Containers & Isolation
Linux - Security
Linux - eBPF
Observability & Tracing
Filesystems & Storage
Linux - Syscalls
Linux - Versus