gVisor - LISAFS

LISAFS (Linux Sandbox File System) is a custom, high-performance file system protocol developed by the gVisor team. It was designed specifically to replace the older 9P protocol used for communication between the Sentry (the sandbox) and the Gofer (the file proxy).

If 9P was a "good enough" generic solution, LISAFS is the "highly tuned" replacement built for speed.

Why was LISAFS created? (The 9P Problem)

9P is very "chatty." To perform a simple operation like cat /etc/config.json, the Sentry and Gofer have to exchange several messages:

Sentry: "Walk to /etc" => Gofer: "Done"
Sentry: "Walk to config.json" => Gofer: "Done"
Sentry: "Open it" => Gofer: "Here is the handle"
Sentry: "Read 4KB" => Gofer: "Here is the data"

Every one of these round-trips incurs a performance penalty. In a high-density environment like Kubernetes, these milliseconds add up, making file-heavy applications (like web servers or build tools) feel sluggish.

The Core Innovation: Control Path vs. Data Path

LISAFS splits communication into two distinct "lanes" to maximize efficiency:

The Control Path (The "Brain")

For things like "Create a directory," "Delete a file," or "Change permissions," LISAFS uses a Unix Domain Socket. These are discrete commands that happen relatively infrequently.

Optimization: LISAFS allows for batching. Instead of sending three messages to navigate a directory tree, the Sentry can send one "WalkPath" request that handles multiple levels at once.

The Data Path (The "Muscle")

For reading and writing actual file contents (the heavy lifting), LISAFS uses Shared Memory.

The Sentry and Gofer set up a shared memory region.
When the application wants to read a file, the Gofer reads the data from the host disk and writes it directly into that shared memory.
The Sentry reads it from the same memory.

Result: Zero-copy data transfer. The data never has to be serialized into a socket and "copied" back and forth.

Key Technical Features

Linux-Native Semantics: 9P was designed for the Plan 9 OS, so gVisor had to "translate" Linux concepts into 9P concepts. LISAFS is designed specifically for Linux. If a Linux syscall asks for a specific file attribute, LISAFS has a field for that exact attribute. No translation layer means less CPU work.
Ring Buffers: LISAFS uses "Ring Buffers" in shared memory for its request/response queue. This allows the Sentry and Gofer to communicate without needing to constantly wake each other up with expensive "interrupts" (context switches) if they are both busy.
Handle Caching: LISAFS is much more aggressive about caching file metadata. The Sentry can keep track of file sizes and permissions longer, reducing the number of times it has to ask the Gofer for updates.

LISAFS vs. 9P: A Comparison

Feature	9P (The Old Way)	LISAFS (The New Way)
Protocol	Generic (Plan 9)	Linux-Specific
Communication	Unix Domain Socket only	UDS + Shared Memory
Data Transfer	Copying (Slower)	Zero-copy (Much Faster)
Round-trips	High (Chatty)	Low (Batched)
Efficiency	Moderate	High (optimized for high-I/O)

Do we still need SCM_RIGHTS?

The move to LISAFS (and shared memory) significantly changes how the Sentry and Gofer communicate, but it does not entirely eliminate the need for SCM_RIGHTS.

Think of it this way: Shared Memory is for moving the payload (the data), while SCM_RIGHTS is for moving the permission (the handle).

Bootstrapping the Shared Memory itself

This is the most ironic part: You need SCM_RIGHTS to set up the Shared Memory.

In Linux, shared memory is often created using memfd_create. This gives you a File Descriptor.

The Gofer creates the memory-file (the buffer).
The Gofer has the FD for that memory.
The Sentry needs that FD to "map" the memory into its own space.
How does the Gofer give that FD to the Sentry? SCM_RIGHTS.

So, at a minimum, SCM_RIGHTS is the "key" that opens the shared memory door.

The "Host FD" Optimization (Bypassing LISAFS)

LISAFS is fast, but it is still a protocol (Request → Response). For some high-performance workloads, gVisor uses an optimization called Host FD Donation.

If the Sentry decides a file is being accessed very frequently, it will ask the Gofer: "Don't just send me the data over LISAFS; give me the actual Host File Descriptor."

The Gofer sends the FD via SCM_RIGHTS.
The Sentry can now call read() and write() directly on that FD.
This completely bypasses the LISAFS protocol and the shared memory buffers, reaching near-native speeds.

Memory Mapping (`mmap`)

If an application inside the container calls mmap() (to map a file directly into its memory, common for databases or loading libraries):

gVisor cannot easily "emulate" a real kernel mmap through a LISAFS shared memory buffer.
The Sentry needs the actual Host FD for that specific file so it can ask the Host Kernel to map that physical file into the Sentry’s address space.
To get that handle, it must use SCM_RIGHTS.

How LISAFS + Shared Memory Changed the Game

Before LISAFS (when gVisor used the 9P protocol), it was much more dependent on SCM_RIGHTS for everything.

With LISAFS:

For standard files: gVisor prefers the Shared Memory Ring Buffer. It doesn't pass a new FD for every single open(). Instead, it uses a persistent connection and moves data through the shared RAM. This is much faster because SCM_RIGHTS (and the kernel intervention it requires) is relatively "expensive."
The SCM_RIGHTS "Demotion": It went from being the daily worker to being the specialist. It is now used primarily for setup, for mmap, and for "donating" high-performance handles.

Summary Table

Feature	LISAFS (Shared Memory)	SCM_RIGHTS (FD Passing)
Purpose	Moving the bytes of the file.	Moving the identity of the file.
Efficiency	Extremely fast (No kernel calls to move data).	Slower (Requires Kernel intervention).
Used For	Regular `read` / `write` operations.	`mmap`, Setup, and Direct-Host-IO.
Analogy	The conveyor belt that carries the coal.	The key that opens the factory gate.

Is SCM_RIGHTS more expensive than copying file to the shared memory?

Yes, SCM_RIGHTS is significantly more expensive to "execute" than a single memory copy.

Why `SCM_RIGHTS` is Expensive (The Kernel Tax)

When you use SCM_RIGHTS to pass a File Descriptor, you are asking the Linux Kernel to do a massive amount of "bureaucracy."

During SCM_RIGHTS, the Kernel must:

Context Switch: Transition from User Mode to Kernel Mode.
Parse the Message: Scan the ancillary data to find the FD.
Validate: Check if the sender actually owns that FD.
Allocate: Find the first available integer in the receiver's private FD table.
Clone: Duplicate the kernel's internal struct file pointer.
Reference Counting: Increment the "open count" of the file so it isn't deleted while being passed.
Cleanup: Manage the lifecycle of the message in the socket buffer.

This is a heavyweight operation. If you had to do SCM_RIGHTS every time you wanted to read 4KB of data, your performance would be terrible.

Why Shared Memory (LISAFS) is Cheap

Shared memory is the fastest way for two processes to talk because the Kernel is not involved.

During a Shared Memory move:

The Move: Process A writes data to a memory address.
The Read: Process B reads from that same address.
Kernel Involvement: Zero. There are no syscalls, no context switches, and no "cloning" of structures. It is just electrons moving through RAM at the speed of the memory bus.

Note: You still need a tiny notification (like an eventfd) to say "Hey, data is ready," but that is much lighter than passing a full file descriptor.

The Trade-off: One-time Cost vs. Every-time Cost

This is the key to understanding why gVisor uses both.

The LISAFS Strategy (Shared Memory)

Best for: Standard, small, or frequent read/write calls.
Cost: You pay a small "protocol" cost to ask for the data, but the data move itself is nearly free.
The Downside: The data still has to be "copied" from the Host's kernel buffer into the Shared Memory buffer, and then potentially copied again into the App's buffer.

The SCM_RIGHTS Strategy (Direct FD Donation)

Best for: Large files, mmap, or extremely I/O-heavy applications.
Cost: You pay a huge "entry fee" (the SCM_RIGHTS syscall) to get the FD.
The Reward: Once the Sentry has that FD, it can talk directly to the Host Kernel. It no longer needs to talk to the Gofer at all for that file.
Zero-Copy: The Sentry can tell the host kernel: "Read this file directly into the App's memory." This eliminates the Gofer and the shared memory middle-man entirely.

Why doesn't gVisor just use `SCM_RIGHTS` for everything?

If SCM_RIGHTS allows direct host access, why not use it for every file?

Security (The "Blast Radius"): If the Sentry has a direct Host FD, it is "closer" to the host kernel. gVisor prefers to keep the Gofer in the middle as a shield whenever possible.
Resource Limits: Every open FD consumes kernel memory. If a container opens 100,000 small files, performing SCM_RIGHTS 100,000 times would overwhelm the host kernel's process table and drastically slow down the system.
The "Sweet Spot": gVisor uses LISAFS by default because it's the best balance of security and speed for typical web apps. It only "upgrades" to SCM_RIGHTS when it absolutely has to (like for mmap) or when it detects a high-performance need.

Conclusion: SCM_RIGHTS is the most expensive "handshake" in Linux, but once that handshake is done, the resulting connection is the fastest possible path.