Building a CSI Driver with Go

A hands-on guide, one gRPC call at a time

How this book works

Every chapter adds one working piece to a real CSI driver — no chapter is theory-only. Starting in Chapter 3, every method is written test-first (red, green, refactor), and design decisions are pointed out through the SOLID principles at the exact point they apply, rather than taught as separate theory. By the end you'll have a driver that provisions, mounts, and snapshots volumes on a kind cluster, backed by nothing more exotic than local directories on the node (so you can run every example without cloud credentials). The RPC signatures — what CreateVolume, NodePublishVolume, and the rest take and return — are exactly what a real EBS or NAS-backed driver would implement too; what's different is everything inside each method: attach/detach against a real API, durable metadata, topology, and backend-specific failure handling all have to be built out for real, not swapped in from one file.

Worth naming honestly up front: later chapters mostly show edits to files a previous chapter already built — "replace the old return with this," "add this method below" — rather than reprinting a complete file every time. That keeps each chapter's code block focused on the one thing it's teaching, but it also means there's no single "final state" file to compare against if you get lost. The fix is cheap and worth doing as a habit, not a one-time setup step: after applying a chapter's edits, run

gofmt -l . && go vet ./... && go build ./... && go test ./...

before moving on. Every chapter in this book holds to exactly that bar before its own code block is considered finished — if any of those four commands complain, something from that chapter didn't get applied the way it was written, and it's much easier to find on a chapter boundary than three chapters later.

Table of contents

1. Why a CSI Driver — what CSI actually is, the three gRPC services, why Kubernetes offloads storage this way
2. Set Up the Project — Go module, directory layout, protobuf/gRPC toolchain, kind cluster 2.5. A Brief gRPC Primer — the request/response model, why CSI uses Unix sockets instead of TCP, how spec error codes become Go code
3. The Identity Service — the simplest of the three services; get it running end-to-end over a Unix socket before touching anything else
4. Registering with Kubelet — the node-driver-registrar sidecar, CSIDriver object, deploying your first pod
5. The Node Service, Part 1 — NodeGetInfo, NodeGetCapabilities
6. The Node Service, Part 2 — NodePublishVolume / NodeUnpublishVolume: actually mounting something into a pod
7. The Controller Service — CreateVolume / DeleteVolume, idempotency, the external-provisioner sidecar
8. Attaching Volumes — ControllerPublishVolume, why this single-node driver doesn't deploy the external-attacher sidecar, StorageClass and PVC end to end
9. Testing with csi-sanity — the official conformance suite, what it catches that manual testing won't
10. Snapshots — CreateSnapshot, external-snapshotter, VolumeSnapshot objects
11. Getting to Production — structured error codes, observability (structured logging, Prometheus metrics served for real over HTTP)
12. Integration Testing — automated Go tests against a real kind cluster, codifying the manual kubectl/grpcurl proofs this book has done by hand since Chapter 8, including a regression test for the real CreateSnapshot idempotency bug Chapter 10 hit live

Each chapter ends with a working git commit-able state. Chapters 3–8 are the core; 9–10 round the driver out to something conformant; 11 is about what changes when this stops being a toy; 12 closes the loop by automating the real-cluster verification this book has relied on by hand throughout.

Cut from the original plan: Volume Expansion (ControllerExpandVolume/ NodeExpandVolume) doesn't teach much for this driver — CreateVolume never enforced real capacity limits (just recorded a number in metadata), so "expanding" a local directory has no real backing mechanism to exercise. Left out rather than padded with a hollow chapter.

Chapter 1: Why a CSI Driver

The problem CSI solves

Before 2019, if you wanted Kubernetes to talk to a new kind of storage, you wrote your driver inside Kubernetes itself — in the k8s.io/kubernetes tree, in Go, released on Kubernetes's schedule. Your storage bug fix waited for the next Kubernetes minor version. Your driver shipped with every cluster whether anyone used it or not.

The Container Storage Interface (CSI) is Kubernetes's answer: a small, language-agnostic gRPC contract that lives outside core Kubernetes. You implement the contract, ship your driver as its own container image, and Kubernetes talks to it over gRPC. Your release cycle is yours. Kubernetes doesn't need to know your storage system exists — it only needs to know your driver speaks CSI.

That's the whole idea. Everything else in this book is working out the mechanics of "speaks CSI."

The three services

A CSI driver implements up to three gRPC services. You don't have to implement all of them in one binary — many drivers ship the Controller service in one Deployment and the Node service in a DaemonSet — but conceptually there are three jobs:

• Identity — "who are you, what can you do?" Every driver implements this. It's how the sidecars (more on those in a moment) discover your driver's capabilities before calling anything else.
• Controller — the cluster-wide brain. Creates and deletes volumes, attaches and detaches them from nodes, takes snapshots. Runs once (or as a leader-elected few) per cluster, because "create a volume" is a cluster-level decision, not a per-node one.
• Node — runs on every node as a DaemonSet. Once a volume exists and is attached to a node, the Node service is what actually mounts it into a pod's filesystem, and unmounts it when the pod goes away.

Here's the mental model that makes the split click: Controller decides, Node executes. In the common case, the Controller service handles provisioning and attachment while the Node service handles mounting — this driver's own Controller happens to touch the filesystem too, because "provisioning" for a local-directory backend just is creating a directory, but a real network-storage driver's Controller would only ever call its backend's API, never touch a filesystem directly. Either way, the Node service's job stays the same: run mount on whatever the Controller already made sure exists and is attached to the local node, never talk to the backend's API itself.

Why gRPC, and why that matters to you

CSI is defined as .proto files — Protocol Buffer service definitions. This is worth sitting with for a second, because it explains a lot about how you'll spend your time in this book: you are not going to design an API. The API is already designed, by the CSI spec, and it doesn't change based on what storage backend you're writing. Your job is to implement Go functions with signatures that already exist.

That's a very different exercise from typical Go service development. You won't be deciding "what should CreateVolume return?" — the spec already decided. You'll be deciding "given what CreateVolume must return, how do I make that true for my storage backend?"

This also means the official csi Go package — generated from those same .proto files — gives you every request/response struct, every error code, before you write a line of driver logic. We'll pull it in next chapter.

Sidecars: the part nobody explains well

New CSI developers often get confused here, so let's be explicit: your driver never talks to the Kubernetes API server directly. You don't watch for PersistentVolumeClaims. You don't create PersistentVolume objects. A set of Kubernetes-maintained sidecar containers does that translation for you:

• external-provisioner watches PVCs, and when one needs a volume, calls your Controller service's CreateVolume over a shared Unix socket.
• external-attacher watches VolumeAttachment objects and calls your Controller's ControllerPublishVolume.
• node-driver-registrar tells kubelet "here's a CSI driver at this socket path, here's its name" by calling your Identity service.
• external-resizer and external-snapshotter do the equivalent for expansion and snapshots.

Each sidecar is a small, boring, well-tested Go program maintained by kubernetes-csi on GitHub. You never write one. You run them as extra containers in the same pod as your driver, sharing a Unix socket over an emptyDir volume. Your driver's entire job is to answer the gRPC calls those sidecars make.

This is why, when you go looking at a CSI driver pod later in this book, you'll see four or five containers in it and only one of them is "yours."

What "idempotent" means here, concretely

The spec requires most RPCs to be idempotent, and this isn't a nice-to-have — sidecars will retry. CreateVolume might get called twice for the same name because the provisioner didn't see your first response before a restart. Your driver has to recognize "I already created a volume for this name" and return the existing volume instead of creating a second one or erroring.

We'll hit this for real in Chapter 7, but it's worth previewing now because it shapes how you should read the spec: every RPC's doc comment is really answering two questions — "what does this do?" and "what does this do if it's called again with the same arguments?" Both matter equally.

What you'll have by the end of this chapter

Nothing runnable yet — that starts next chapter. But you should now be able to answer, without looking anything up:

• What problem CSI solves that in-tree drivers didn't
• The job of each of the three services, and the "Controller decides, Node executes" split
• Why you'll never see your driver call the Kubernetes API directly
• Why idempotency isn't optional

Chapter 2 sets up the project: Go module, the csi package, and a kind cluster to deploy into. By the end of Chapter 3 you'll have a driver answering its very first real gRPC call over a Unix socket; Chapter 4 is where it actually gets deployed into that cluster and registered with kubelet.

Chapter 2: Set Up the Project

We're building localdir-csi — a driver that provisions volumes as plain directories on the node's local disk. It's not a driver you'd run in production (no real attach/detach story, no replication), but every RPC you implement here has the exact same signature a real EBS or NFS driver would use. When you're done, the RPC contract carries over — the actual work of retargeting CreateVolume at a real storage API, with durable metadata and real attach/detach, does not.

Prerequisites

You'll need:

• Go 1.24+ (tests from Chapter 3 onward use t.Context(), a method added to the testing package in Go 1.24 — more on why in Chapter 3)
• Docker (or another kind-compatible container runtime)
• kind, kubectl

Install kind if you don't have it:

go install sigs.k8s.io/kind@latest

The project skeleton

mkdir localdir-csi && cd localdir-csi
go mod init github.com/yourname/localdir-csi

Layout we're building toward — don't create all of this now, just know where things are headed:

localdir-csi/
├── cmd/
│   └── localdir-csi/
│       └── main.go          # entrypoint: parse flags, start gRPC server
├── internal/
│   └── driver/
│       ├── driver.go         # Driver struct, shared by all three services
│       ├── identity.go       # Identity service RPCs
│       ├── controller.go     # Controller service RPCs
│       └── node.go           # Node service RPCs
├── deploy/
│   ├── csidriver.yaml
│   ├── controller.yaml       # Deployment: driver + provisioner + attacher sidecars
│   └── node.yaml             # DaemonSet: driver + registrar sidecar
├── go.mod
└── go.sum

Every RPC method you write goes on Driver in internal/driver/. One struct, three interfaces satisfied (csi.IdentityServer, csi.ControllerServer, csi.NodeServer) — that's the whole shape of a CSI driver.

Pulling in the CSI spec package

The CSI project publishes the generated Go types and gRPC service interfaces so you never write .proto files yourself:

go get github.com/container-storage-interface/spec/lib/go/csi
go get google.golang.org/grpc

Confirm it resolved:

go list -m github.com/container-storage-interface/spec

Everything you implement in this book is satisfying interfaces from that one package. Worth opening it once now to see what's there:

go doc github.com/container-storage-interface/spec/lib/go/csi.IdentityServer

You should see three method signatures — GetPluginInfo, GetPluginCapabilities, Probe. That's the entire Identity service. We implement it in the next chapter.

The kind cluster

CSI drivers are the kind of thing that's genuinely painful to develop against a real cloud cluster — every iteration means pushing an image somewhere reachable. kind runs Kubernetes in Docker containers on your laptop, and critically, its nodes are real Linux containers with a real kubelet, so mount calls in your Node service behave exactly like they would on a cloud VM.

kind create cluster --name csi-dev
kubectl cluster-info --context kind-csi-dev

One kind-specific thing worth knowing now, because it'll save you confusion in Chapter 4: kind nodes are containers, so when your Node service later runs mount --bind, it's bind-mounting inside that container's filesystem, not your laptop's. That's fine and expected — it's exactly how a real node's kubelet and CSI node plugin relate — but if you go looking for the volume directories on your actual machine, you won't find them. You'll docker exec into the kind node to look, and we'll do that together in Chapter 6.

Loading local images into kind (you'll use this constantly once we start building the image in Chapter 4):

docker build -t localdir-csi:dev .
kind load docker-image localdir-csi:dev --name csi-dev

A Makefile for the repeated stuff

You'll run the same handful of commands dozens of times over the next ten chapters — build the image, load it into kind, apply manifests, tear the cluster down and stand up a fresh one when state gets weird. Worth wrapping those now rather than retyping them:

CLUSTER  := csi-dev
IMAGE    := localdir-csi:dev

.PHONY: kind-up kind-down build load deploy logs clean

kind-up:
	kind create cluster --name $(CLUSTER)

kind-down:
	kind delete cluster --name $(CLUSTER)

build:
	docker build -t $(IMAGE) .

load: build
	kind load docker-image $(IMAGE) --name $(CLUSTER)

deploy: load
	kubectl apply -f deploy/
	kubectl rollout restart deployment/localdir-csi-controller daemonset/localdir-csi-node 2>/dev/null || true
	kubectl rollout status deployment/localdir-csi-controller --timeout=60s 2>/dev/null || true
	kubectl rollout status daemonset/localdir-csi-node --timeout=60s 2>/dev/null || true

logs:
	kubectl logs -l app=localdir-csi -c localdir-csi --tail=100 -f

clean: kind-down
	rm -f deploy/*.generated.yaml

Save that as Makefile in the project root. A couple of things worth noting about it, since they'll matter as the book goes on:

• load depends on build, and deploy depends on load — make deploy alone rebuilds the image, reloads it into the cluster, and applies manifests, in the right order, every time. No more forgetting you edited node.go and re-applying stale YAML against an old image.
• Rebuilding and reloading the image is not, by itself, enough to make a running pod pick up your change. Every rebuild reuses the same tag, localdir-csi:dev — that's deliberate, so you never have to edit YAML just to bump a tag — but Kubernetes only pulls or restarts a container when something about its spec changes, and an unchanged tag string looks unchanged to it even though the bytes behind it in kind's local image store are new. A pod that was already running keeps running whatever image it already pulled. kubectl rollout restart is the real fix for exactly this: it forces the Deployment/DaemonSet to replace their pods, and the replacements pull the tag fresh from kind's image store — where the bytes you just loaded are now waiting. The || true on all three lines exists because before Chapter 4 there's no deployment/ or daemonset/ for these commands to find yet; once they exist, make deploy always leaves both the Controller and every Node pod running the binary you just built, not whatever was running before.
• deploy/ doesn't exist with real manifests yet — that starts in Chapter 4. The target is here now so the Makefile grows with the book instead of getting rewritten later.
• logs filters to the driver container specifically (-c localdir-csi), because once sidecars are in the picture in Chapter 4, kubectl logs on a pod with five containers is useless without -c.

From here on, when a chapter says "build and deploy," it means make deploy.

What you should have now

• go.mod with the csi and grpc packages resolved
• A kind cluster named csi-dev up and reachable via kubectl (or run make kind-up)
• A Makefile with build / load / deploy / logs targets
• An empty cmd/ and internal/driver/ ready for Chapter 3

Next chapter we write the first real code: a Driver struct, a gRPC server listening on a Unix socket, and the three-method Identity service — small enough to finish in one sitting, and the first point where you can actually run something and watch it answer a gRPC call.

Chapter 2.5: A Brief gRPC Primer

Just enough to make Chapter 3 make sense — not a full gRPC book. This chapter goes slower than the rest of the book, on purpose. Everything from Chapter 3 onward assumes these ideas feel comfortable, so it's worth sitting with them properly now, even if some of this feels obvious.

Start with something you already know: calling a function

Before "remote," before "procedure call," before any of the jargon — start with the plainest possible thing: calling a function in a normal Go program.

func Add(a, b int) int {
    return a + b
}

func main() {
    result := Add(2, 3)
    fmt.Println(result) // 5
}

When main calls Add(2, 3), here's what actually happens, mechanically: your program jumps to the place in memory where Add's code lives, hands it 2 and 3, Add runs, and control jumps back to main with 5 in hand. This works because Add and main are part of the same running program, sitting in the same computer's memory. Jumping to Add's code is cheap and instant — it's all right there.

That's a "procedure call" — "procedure" is just an old, language-neutral word for "function." Nothing remote about it yet.

Now the problem: what if the function lives somewhere else?

Here's the situation CSI actually puts you in. Your driver is one running program. The thing that wants to ask it "what's your name and version?" (GetPluginInfo, which we'll build in Chapter 3) is a different running program — a sidecar container, possibly even on a different machine entirely.

You can't just write driver.GetPluginInfo() from the sidecar's code the way main called Add() above. There's no shared memory to jump into — the sidecar's program and your driver's program are two separate processes that don't know anything about each other's internals. All they can do is send each other bytes over some connection (a network, or, as we'll get to below, a file acting like a pipe).

So the real question is: how do you make something that feels like calling a function — you write driver.GetPluginInfo(), you get a normal return value back — when the actual function is running in a different program, maybe on a different machine, and the only way to reach it is by sending bytes back and forth?

That question, and the standard answer to it, is what "RPC" is.

What RPC actually means, word by word

RPC stands for Remote Procedure Call. Read it backwards, it explains itself:

• Call — you're calling a function, same as Add(2, 3) above.
• Procedure — the function you're calling.
• Remote — except this time, the function isn't in your program. It's somewhere else — another process, another machine.

RPC is a general pattern (not tied to Go, or to gRPC specifically, or even to any one programming language) for making a call to a remote function feel like calling a local one, even though under the hood, something much more involved has to happen: your arguments need to be packaged up, sent across a connection, unpacked and run on the other side, and the result needs to make the same trip back.

An analogy that makes the mechanics concrete

Think about ordering food at a restaurant through a waiter.

You (the calling code) don't walk into the kitchen and cook your own burger. You tell the waiter "I'd like a burger." The waiter doesn't cook either — they write your order down, walk it to the kitchen, and hand it to the chef. The chef (the real function) does the actual work — cooks the burger — and hands the finished plate back to the waiter, who carries it back and puts it in front of you.

From where you're sitting, you asked for a burger and a burger appeared. You never had to know anything about the kitchen, the chef, or how the order got there. The waiter is what made that illusion possible.

RPC's "stub" plays the exact role of that waiter, on both ends:

1. Your code calls what looks like a normal local function — this fake local function is called the client stub. It exists purely to play along with the illusion.
2. Instead of running any real logic, the client stub packages up your arguments into a message and sends that message across a connection to the other program. Packaging data into a format that can be sent as raw bytes is called serializing it (turning a Go struct in memory into a flat sequence of bytes that can travel over a wire); the receiving side reverses this — deserializing — to turn those bytes back into a usable value.
3. On the other side, a matching piece of code — the server stub — receives those bytes, deserializes them back into arguments, and calls the real function, the one that actually does the work (in our case, a method on your Driver struct).
4. The real function runs, produces a return value, and the whole thing happens in reverse: the server stub serializes the result, sends it back across the connection, and the client stub on your side deserializes it and hands it back to your code as an ordinary return value.

Your code never sees any of steps 2 through 4. It just called a function and got a value back — same experience as Add(2, 3), even though a network round trip happened in between. That's the whole point of RPC: hide the network behind something that looks like an ordinary function call.

So what is gRPC, then?

RPC is the idea — "make a remote call look local." It doesn't say exactly how you describe what functions exist, exactly how arguments get turned into bytes, or exactly what carries those bytes from one program to the other. Those are implementation details, and over the decades, lots of different specific systems have implemented the RPC idea in different ways.

gRPC is one specific, concrete implementation of the RPC idea, built by Google (the lowercase "g" has meant different things across Google's own documentation over the years — not worth memorizing, just know gRPC is a specific product, not a generic term). gRPC answers the two questions RPC leaves open:

Question 1: how do you describe, precisely, what functions exist and what their inputs and outputs look like?

gRPC's answer is a language called Protocol Buffers (nicknamed "protobuf"), used to write .proto files. A .proto file describes a function signature the same way a Go interface would, but in a way that isn't tied to any one programming language:

service Identity {
  rpc GetPluginInfo(GetPluginInfoRequest) returns (GetPluginInfoResponse);
}

Read that plainly: "there's a service called Identity. It has one function, GetPluginInfo. You call it with a GetPluginInfoRequest and you get back a GetPluginInfoResponse." That's it — it's just a function signature, written in a language-neutral way so that a Go program and a Python program could both agree on exactly what this function looks like, without either one needing to understand the other's syntax.

A separate tool, protoc (the "protobuf compiler"), reads that .proto file and generates real, runnable code in whatever language you need — Go, in our case. That generated code includes the client stub and the server-side interface described above, already written for you:

type IdentityServer interface {
    GetPluginInfo(context.Context, *GetPluginInfoRequest) (*GetPluginInfoResponse, error)
}

This is exactly the "server stub" idea from the waiter analogy, except you never hand-write it — protoc generates it from the .proto file. Your only job is to write a Go struct with a GetPluginInfo method matching that signature; gRPC and the generated code handle everything about turning network bytes into a Go function call and back.

One thing worth knowing now, because it'll save confusion later: for this book, you will never touch a .proto file or run protoc yourself. The CSI project already did that step. The container-storage-interface/spec Go package you pulled in last chapter is the already-generated code — the IdentityServer, ControllerServer, and NodeServer interfaces are sitting there, ready to import, the same way IdentityServer above was generated from the .proto snippet. Your job starts one level in from where most gRPC tutorials start: write the struct, implement the methods the interface already demands.

Question 2: once your arguments are serialized into bytes, what actually carries those bytes from one program to the other?

gRPC's answer is HTTP/2. You've probably used HTTP before, even if only by typing a URL into a browser — HTTP is the protocol (a shared, agreed-upon set of rules) that web browsers and web servers use to talk to each other. HTTP/2 is a newer, more efficient version of that same protocol, and gRPC uses it as its "carrier" — the actual mechanism moving your serialized bytes from the client stub, across a connection, to the server stub. You don't need to know HTTP/2's internals for this book; just know it's the transport gRPC rides on, playing the same role a road plays for a delivery truck. The truck (your serialized message) doesn't care much about the road's engineering — it just needs a reliable way to get from A to B.

Why gRPC, specifically, and why CSI picked it

Two reasons matter for what you're building:

• Language independence. The sidecars (external-provisioner, node-driver-registrar, and friends) are Go binaries maintained by kubernetes-csi, but your own driver doesn't have to be written in Go — CSI drivers exist in Rust, Python, C++, and more. This works because the actual contract is the .proto file, not any particular programming language. Any language with a protobuf/gRPC library can play.
• A real, typed, checked contract, up to a point. Because the .proto file spells out exactly what fields exist and what types they are, both sides get that checked automatically — in Go's case, at compile time, before your program even runs. "Which fields are required," though, is a different, weaker guarantee: CSI's spec marks fields REQUIRED in comments and in spec.md's own prose, not in a way protobuf enforces. Generated Go code will happily construct a request with an empty Name or a nil VolumeCapability — the compiler has no objection. Enforcing "required" is exactly the job this book's own handler code takes on starting next chapter, and it's also why csi-sanity (Chapter 9) exists as a separate conformance suite: compile- time types catch "wrong shape," not "missing what the spec demands." Compare that to a typical REST API (an HTTP API returning JSON, without gRPC), where "what fields does this endpoint actually require" often lives only in documentation that may or may not be accurate or current — protobuf at least gives you the shape for free; CSI still needs runtime checks for the rest.

One more piece: how the bytes actually travel — sockets

Zoom in one more level on "a connection" from the waiter analogy. In networking, the mechanism two programs use to open a connection and pass bytes back and forth is called a socket. Think of a socket as an address plus a plumbing hookup: something you can "connect to," after which bytes can flow in both directions until one side closes the connection.

Almost every gRPC tutorial you'll find sets up a socket as a TCP socket — an address made of an IP address and a port number, like localhost:50051. This is what lets two programs talk over an actual network, potentially across the internet.

CSI doesn't do that. Instead, the driver and its sidecars talk over a Unix domain socket — which, despite having "socket" in the name and working the same way from your code's point of view, isn't a network address at all. It's a special kind of file that lives on disk (typically at the path /csi/csi.sock), and two programs on the same machine can open it and use it to exchange bytes, the same way they'd use a TCP socket, just without any actual network involved.

Why CSI does it this way: your driver container and the sidecar containers (external-provisioner, node-driver-registrar, etc.) all run inside the same Kubernetes pod, sharing a directory via something called an emptyDir volume. Since they're already on the same machine and sharing a filesystem, there's no reason to expose your driver over the network at all — doing so would mean anything else in the cluster could potentially try to call your Controller service's CreateVolume, with nothing checking who's asking. A file-based socket sidesteps that entirely: only programs that can already see that file on disk (i.e., containers in that same pod) can connect to it. The socket file itself is the access boundary — nothing about the CSI RPCs adds authentication on top of that.

In Go, switching from a TCP socket to a Unix domain socket changes exactly one line versus what most tutorials show you:

// what most gRPC tutorials show — a TCP socket, a network port:
lis, err := net.Listen("tcp", ":50051")

// what CSI drivers actually do — a Unix domain socket, a file path:
lis, err := net.Listen("unix", "/csi/csi.sock")

Everything else about setting up the server — grpc.NewServer(), registering your service, calling Serve() — is identical either way. We write this for real in Chapter 3.

Putting the whole round trip together

Worth having the entire path in your head, plainly, before we build it. Say a sidecar calls GetPluginInfo:

1. The sidecar (or, when you're testing by hand, a tool called grpcurl, which we'll use in Chapter 3) opens a connection to your Unix socket file — same idea as the waiter approaching your table.
2. It calls GetPluginInfo, handing over a request. Behind the scenes, the client stub (generated from the .proto file, remember) serializes that request into protobuf bytes and sends it over the socket, carried by HTTP/2.
3. Your server deserializes those bytes back into a *csi.GetPluginInfoRequest Go struct and hands it to your Driver struct's GetPluginInfo method — the real function, the "chef."
4. Your method does its (very small, in this case) work and returns a *csi.GetPluginInfoResponse and a Go error — completely ordinary Go return values, nothing gRPC-specific about writing this part.
5. If that error is non-nil, gRPC needs a way to communicate what kind of failure happened — not just "something went wrong," but which of a known, fixed set of reasons. So gRPC translates your Go error into a gRPC status code: a fixed enum of reasons like OK, NotFound, AlreadyExists, InvalidArgument, and so on (this is a gRPC-specific concept, a different list from HTTP status codes like 404 or 500, though it plays a similar role). The CSI spec is written in terms of exactly these codes — it says things like "this method MUST return ALREADY_EXISTS" when a volume by that name already exists — so this status-code mechanism is what turns a plain-English spec requirement into real, checkable Go code, almost always written as status.Error(codes.AlreadyExists, "...") using the google.golang.org/grpc/status package. You'll type that pattern constantly starting next chapter.
6. The response (or the error, translated into its status code) gets serialized and sent back over the same socket to the sidecar, which deserializes it and — from its point of view — just got a return value from a function call. The illusion holds all the way through.

One thing to not worry about yet

Streaming RPCs (calls that send many messages over time instead of one request/one response), interceptors (middleware that runs before or after every call), TLS (encrypting the connection), and deadlines/cancellation propagation (telling a slow call to give up early) are all real gRPC concepts — none of them are needed to get a working CSI driver's happy path running. Every RPC in CSI is a simple unary call: one request goes out, one response comes back, exactly like the GetPluginInfo walkthrough above. We'll mention deadlines briefly when we get to Probe in Chapter 3. Interceptors turn out not to stay off this list either — Chapter 11 comes back to them once there's a real operational reason (logging, metrics) to want code that runs before and after every RPC. Streaming and TLS never come up again; this driver has no need for either.

That's the whole primer. Chapter 3 builds directly on top of every piece of it — the stub idea, protobuf, sockets, status codes — starting with writing the Driver struct and getting the Identity service answering real calls over that Unix socket.

Chapter 3: The Identity Service

This is the first chapter where you'll run real code and see it answer a real question over the network. We're going to keep it small on purpose — three methods, none of which touch storage at all — so you can focus entirely on "how does a CSI driver's code fit together" without also worrying about mounts, volumes, or the Kubernetes API.

What the Identity service is for, in plain words

Before Kubernetes trusts your driver to create or mount anything, it wants to ask it two simple questions: "who are you?" and "are you still alive and working?"

That's it. That's the whole Identity service. It doesn't create volumes. It doesn't mount anything. It just answers questions about itself. Every CSI driver — no matter what storage system it talks to — implements this exact same service, because every driver needs to answer these same two questions.

The three methods, in the plainest terms possible:

• GetPluginInfo — "What's your name and version number?" Kubernetes uses this to tell your driver apart from every other driver installed in the cluster.
• GetPluginCapabilities — "What are you able to do?" Some drivers can only create and delete volumes. Others can also take snapshots, or resize volumes. This method is how your driver tells Kubernetes which of those things it supports, so Kubernetes doesn't try to ask for a snapshot from a driver that never learned how to make one.
• Probe — "Are you healthy right now?" This gets called repeatedly, over and over, for as long as your driver runs. It's a health check.

None of these three methods need to know anything about your storage backend. That's exactly why we're starting here — you get to learn how the plumbing works (the gRPC server, the Unix socket, how a request turns into a Go function call) without also learning how to create a volume at the same time.

A note on how we'll write code from here on

Starting this chapter, every piece of driver code in this book follows two practices, and it's worth explaining both in plain terms once, up front, so the code doesn't feel like it's following rules you can't see.

Test-Driven Development (TDD) means: write a test for a small piece of behavior before you write the code that makes it work. Run the test, watch it fail — you have to actually see it fail, not assume it would — then write the smallest amount of code that makes it pass, then clean that code up if it needs it. This cycle has a name — red, green, refactor — red for the failing test, green for passing, refactor for tidying up once you're safely covered by a passing test. We'll do this for every method in this chapter.

SOLID is five design guidelines for object-oriented code, one letter each. We won't lecture on all five in the abstract — we'll point out each one exactly where it shows up in the driver, so you see the guideline and the reason for it side by side instead of as a separate rule to memorize:

• S — Single Responsibility: a piece of code should have one reason to change.
• O — Open/Closed: you should be able to add new behavior without editing code that already works.
• L — Liskov Substitution: anything implementing an interface should be swappable for anything else implementing that same interface, with no surprises.
• I — Interface Segregation: don't force code to depend on methods it doesn't use.
• D — Dependency Inversion: code should depend on an interface describing what it needs, not on one specific concrete implementation of it.

Two of these are already true of the project without us doing anything: Interface Segregation is baked into the CSI spec itself — IdentityServer, ControllerServer, and NodeServer are three separate Go interfaces, not one giant interface with every method mashed together. Code that only needs to check plugin info never has to know ControllerServer exists. Liskov Substitution falls out of that too — since Driver implements csi.IdentityServer purely through its method signatures, gRPC can hand it to csi.RegisterIdentityServer and it works, no special cases.

The other three — Single Responsibility, Open/Closed, Dependency Inversion — take actual design decisions on our part, and we'll make those decisions below, with tests, as we go.

The Driver struct: one home for everything

Every method you write in this book — whether it belongs to Identity, Controller, or Node — is going to be a method on one shared Go struct called Driver. Right now it'll be nearly empty: just enough to give the tests below something to construct. Later chapters will add fields as the Controller and Node services need somewhere to keep state.

Create internal/driver/driver.go:

package driver

import (
	"github.com/container-storage-interface/spec/lib/go/csi"
)

// Driver holds everything our CSI driver needs to answer gRPC calls.
type Driver struct {
	// Embedding UnimplementedIdentityServer is what the csi package
	// requires from every IdentityServer implementation — more on why,
	// and what it actually does, just below.
	csi.UnimplementedIdentityServer

	name    string
	version string
}

// NewDriver builds a Driver. We pass the name and version in from main.go
// rather than hardcoding them here, so tests can construct a Driver with
// whatever values they like.
func NewDriver(name, version string) *Driver {
	return &Driver{
		name:    name,
		version: version,
	}
}

// Compile-time check: this line does nothing at runtime. It exists purely
// so that if you ever change Driver in a way that breaks the
// IdentityServer interface, the build fails immediately with a clear
// error — instead of failing later, mysteriously, when gRPC tries to
// register your service.
var _ csi.IdentityServer = (*Driver)(nil)

Two things in this file are worth understanding properly, because you'll meet both patterns again for ControllerServer and NodeServer in later chapters.

The embedded csi.UnimplementedIdentityServer is the CSI spec's answer to a real problem: what happens if a future version of the spec adds a fourth method to IdentityServer? Without some kind of safety valve, every existing driver in the world — including this one — would stop compiling the moment it upgraded to that newer version of the csi package, simply because it would no longer satisfy the interface. UnimplementedIdentityServer is that safety valve: it's a struct with a stub version of every method the interface currently requires, each one just returning a gRPC error saying "not implemented." Embed it in Driver, and Driver automatically satisfies IdentityServer — today, and for any method the spec adds later — without you writing a line for it. Go's embedding rules mean those stub methods become part of Driver's own method set, promoted up from the embedded field, exactly as if you'd written them by hand. IdentityServer also requires one unexported method, mustEmbedUnimplementedIdentityServer(), that exists for no reason other than to force this embedding: there's no way to satisfy it except by embedding UnimplementedIdentityServer, so it's not possible to accidentally build an IdentityServer implementation that skipped this safety valve. Forget the embed, and Go tells you exactly that: *Driver does not implement csi.IdentityServer (missing method mustEmbedUnimplementedIdentityServer).

The compile-time check — var _ csi.IdentityServer = (*Driver)(nil) — is a small, well-known Go trick worth learning once, because you'll use it constantly from here on. It declares a variable named _ (Go's "throw this away, I don't need it" name) of type csi.IdentityServer, and assigns it a nil *Driver. Go only allows that assignment if *Driver genuinely satisfies every method csi.IdentityServer requires. Thanks to the embedded stub above, it already does — so this line compiles cleanly right now, even though Driver doesn't have a single real Identity method of its own yet. That's exactly the outcome the embedding is designed to produce: a driver that builds successfully at every stage of being written, always answering honestly about what it can and can't do yet, rather than refusing to compile at all until every method is finished.

What we're testing first

GetPluginInfo is the simplest possible thing to test: given a driver with a certain name and version, does it report that name and version back? No branching, no edge cases yet. Good first target for TDD, because the test itself takes thirty seconds to write and you can focus entirely on the rhythm — red, green, refactor — rather than on a hard problem.

Create internal/driver/identity_test.go:

package driver

import (
	"testing"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func TestGetPluginInfo(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0")

	resp, err := d.GetPluginInfo(t.Context(), &csi.GetPluginInfoRequest{})
	if err != nil {
		t.Fatalf("GetPluginInfo returned an error: %v", err)
	}

	if resp.Name != "localdir.csi.example.com" {
		t.Errorf("Name = %q, want %q", resp.Name, "localdir.csi.example.com")
	}
	if resp.VendorVersion != "0.1.0" {
		t.Errorf("VendorVersion = %q, want %q", resp.VendorVersion, "0.1.0")
	}
}

One thing worth noticing before running this: the request is built with t.Context() as its first argument, not the more commonly seen context.Background(). Every gRPC method — including every method you'll write in this entire book — takes a context.Context as its first parameter, because gRPC uses it to carry two things alongside a call: a signal meaning "give up on this, right now" (cancellation), and, optionally, a point in time meaning "give up on this if it's not done by then" (a deadline). A context.Context doesn't do any work itself — it's just a small object your code can check, at any point, to ask "should I stop what I'm doing?"

context.Background() gives you the plainest possible context: one that never cancels and never has a deadline. It's a fine starting point when nothing else is available, but it isn't tied to anything — nothing about it knows this call happens to be running inside a test.

t.Context(), a method added directly to Go's testing package in Go 1.24, gives you a context that's scoped to the test that requested it: Go cancels it automatically the moment the test finishes — right before any t.Cleanup functions run — whether the test passed, failed, or was stopped early. For these three Identity methods, this makes no visible difference yet, because none of them look at ctx at all. It's still the right default to reach for from here on, though: later chapters add Controller and Node methods that do real work — copying data, running mount — and check ctx.Done() to bail out early if the caller gave up. Using t.Context() now, in tests where it doesn't matter yet, means you never have to remember to switch away from context.Background() later, in a test where it does. Worth being honest about that promise, though: this book's own Controller and Node methods never actually call ctx.Done() — Chapter 11 names that as one of this driver's real, still- open limitations, not something later chapters quietly deliver. Passing ctx through everywhere is table stakes for a method that could honor cancellation; actually checking it is a separate, deliberate piece of work this book doesn't do.

One consequence worth flagging: because t.Context() lives on *testing.T rather than the context package, this test file has no direct use for "context" at all — notice it isn't in the import block above. If you copy this from an older gRPC tutorial that still imports "context" for context.Background(), Go's compiler will flag it as an unused import the moment you switch to t.Context().

Run it:

go test ./internal/driver/...

This is the moment the embedded stub in driver.go actually matters: d.GetPluginInfo already exists — promoted from UnimplementedIdentityServer — so the package builds without complaint. The test runs. And it fails, not because anything failed to compile, but because the stub does exactly the job it's built for:

--- FAIL: TestGetPluginInfo (0.00s)
    identity_test.go:14: GetPluginInfo returned an error: rpc error: code = Unimplemented desc = method GetPluginInfo not implemented
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.002s

That's red — just a different flavor of red than "undefined method." Nothing here is broken; GetPluginInfo genuinely has no real logic yet as far as Driver is concerned, and the embedded stub says so as clearly as a compile error would, just one step later, at test-run time instead of build time. This is the shape every RPC's first red step takes from here on: never "doesn't exist," always a real, honest response your driver actually sent — just not the one the test wants.

Now write just enough to go green. This is internal/driver/identity.go:

package driver

import (
	"context"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func (d *Driver) GetPluginInfo(
	ctx context.Context,
	req *csi.GetPluginInfoRequest,
) (*csi.GetPluginInfoResponse, error) {
	return &csi.GetPluginInfoResponse{
		Name:          d.name,
		VendorVersion: d.version,
	}, nil
}

This method, defined directly on *Driver, takes over from the embedded stub of the same name. Go always prefers a method defined directly on a type over one promoted from an embedded field, so nothing extra is needed to make the override "stick" — the moment this function exists, every call to d.GetPluginInfo(...) reaches it instead of the stub.

Notice identity.go itself still imports "context" — that's expected and different from the test file. The production method's signature has to name the context.Context type so it satisfies csi.IdentityServer, regardless of what kind of context a caller happens to pass in. Only the test file, which only ever constructs a context via t.Context() rather than naming the type, was able to drop the import.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

That's green. Nothing to refactor yet — the code's already about as simple as it can be. We'll hit a real refactor step later in this chapter.

GetPluginCapabilities: the same rhythm, then a design decision

Same pattern — test first:

func TestGetPluginCapabilities(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0")

	resp, err := d.GetPluginCapabilities(t.Context(), &csi.GetPluginCapabilitiesRequest{})
	if err != nil {
		t.Fatalf("GetPluginCapabilities returned an error: %v", err)
	}

	if len(resp.Capabilities) != 1 {
		t.Fatalf("got %d capabilities, want 1", len(resp.Capabilities))
	}

	service := resp.Capabilities[0].GetService()
	if service == nil {
		t.Fatal("expected a Service capability, got something else")
	}
	if service.Type != csi.PluginCapability_Service_CONTROLLER_SERVICE {
		t.Errorf("capability type = %v, want CONTROLLER_SERVICE", service.Type)
	}
}

Red, the same way as before: GetPluginCapabilities currently resolves to the embedded stub, so it comes back with a gRPC Unimplemented error instead of any capabilities at all.

--- FAIL: TestGetPluginCapabilities (0.00s)
    identity_test.go:XX: GetPluginCapabilities returned an error: rpc error: code = Unimplemented desc = method GetPluginCapabilities not implemented
FAIL

Now the implementation:

func (d *Driver) GetPluginCapabilities(
	ctx context.Context,
	req *csi.GetPluginCapabilitiesRequest,
) (*csi.GetPluginCapabilitiesResponse, error) {
	return &csi.GetPluginCapabilitiesResponse{
		Capabilities: []*csi.PluginCapability{
			{
				Type: &csi.PluginCapability_Service_{
					Service: &csi.PluginCapability_Service{
						Type: csi.PluginCapability_Service_CONTROLLER_SERVICE,
					},
				},
			},
		},
	}, nil
}

Green. Here's where Open/Closed becomes a real, concrete concern rather than an abstract rule: by Chapter 10 (snapshots), this method needs to report a second capability. If capabilities were, say, three separate if statements each building and returning early, adding a fourth capability later would mean editing existing, working, already-tested branches — exactly what Open/Closed says to avoid. Because we built it as one slice literal instead, adding capability #2 later means adding an entry to the slice, not changing how capability #1 is built or tested. Nothing about TestGetPluginCapabilities above will need to change when Chapter 10 adds a second entry — it only asserts Capabilities[0], and a new test will assert Capabilities[1]. Open for extension, closed for modification, made real instead of abstract.

Probe: where Dependency Inversion actually earns its keep

This is the one worth slowing down for. It would be easy to make Probe simply return ready: true unconditionally — there'd be nothing to check, so nothing to get wrong. But think about what Probe is for: a real driver might need to say "not ready" if, say, it can't reach its storage backend yet. Our driver's backend is the local disk, so it should say "not ready" if that disk isn't there or isn't writable.

Here's the design question TDD forces you to confront immediately: how do you write a test for "Probe returns not-ready when the disk is unavailable" without actually detaching a real disk in your test suite? You can't, and you shouldn't try — that's a slow, flaky test that has nothing to do with the logic you're actually verifying.

The fix is Dependency Inversion: instead of Probe reaching out and checking the filesystem itself, Driver depends on a small interface that describes "something that can report whether it's healthy" — and the real filesystem check becomes just one implementation of that interface, swapped in at startup. In tests, we swap in a fake instead.

Test first. This test needs Driver to accept a health checker, which doesn't exist yet — write the test as if it already does:

type fakeHealthChecker struct {
	healthy bool
}

func (f *fakeHealthChecker) Healthy() bool {
	return f.healthy
}

func TestProbe(t *testing.T) {
	tests := []struct {
		name    string
		healthy bool
		want    bool
	}{
		{name: "backend is healthy", healthy: true, want: true},
		{name: "backend is unhealthy", healthy: false, want: false},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			d := NewDriver("localdir.csi.example.com", "0.1.0", &fakeHealthChecker{healthy: tt.healthy})

			resp, err := d.Probe(t.Context(), &csi.ProbeRequest{})
			if err != nil {
				t.Fatalf("Probe returned an error: %v", err)
			}

			if resp.GetReady().GetValue() != tt.want {
				t.Errorf("Ready = %v, want %v", resp.GetReady().GetValue(), tt.want)
			}
		})
	}
}

A couple of things about this test worth calling out:

• The fake is handed straight to NewDriver as a third argument — a constructor that doesn't exist yet, on purpose. TDD's "write the test as if it already does" applies to function signatures, not just to whether a method exists: writing the call the way we want it to look, before it compiles, is what drives the very next step.
• fakeHealthChecker is a tiny struct we control completely — we can make it report healthy or unhealthy on command, instantly, with no real disk involved. This is the entire payoff of Dependency Inversion: the test gets to substitute a fake for the real thing because Driver only ever asked for something with a Healthy() bool method, never specifically for "the real filesystem checker."
• This is a table-driven test — a Go convention where instead of writing one test function per case, you write a slice of cases and loop over them with t.Run. It's not required by TDD or SOLID, but it's the idiomatic way to write "same logic, different inputs" tests in Go, and you'll see this shape constantly for the rest of the book.
• Notice t.Context() is called on the inner t — the one belonging to the subtest created by t.Run, not the outer one belonging to TestProbe itself. Each subtest gets its own context, canceled when that specific subtest finishes, rather than sharing one context across every case in the table.

Run it — red, and this time it's a compile error again, not a stub response, because NewDriver itself is the thing that doesn't match yet — there's no embedded stub standing in for a constructor, only for interface methods:

./identity_test.go:XX:13: too many arguments in call to NewDriver
	have (string, string, *fakeHealthChecker)
	want (string, string)

Now make it real. First, the interface, and an update to the Driver struct and NewDriver we wrote earlier in this chapter — edit driver.go:

package driver

import (
	"github.com/container-storage-interface/spec/lib/go/csi"
)

// HealthChecker describes anything that can report whether the driver's
// backend is usable right now. Identity's Probe method depends on this
// interface — not on any specific way of checking health — so that a
// real filesystem check and a test fake are equally valid things to hand
// it. This is the whole idea behind Dependency Inversion: Driver names
// what it needs, not how that need gets satisfied.
type HealthChecker interface {
	Healthy() bool
}

// Driver holds everything our CSI driver needs to answer gRPC calls.
type Driver struct {
	csi.UnimplementedIdentityServer

	name    string
	version string
	health  HealthChecker
}

// NewDriver builds a Driver. health can be nil — we guard against that in
// Probe — which is useful for tests that don't care about health checks,
// but main.go, wired up later in this chapter, always passes a real
// implementation.
func NewDriver(name, version string, health HealthChecker) *Driver {
	return &Driver{
		name:    name,
		version: version,
		health:  health,
	}
}

// Compile-time check: this line does nothing at runtime. It exists purely
// so that if you ever change Driver in a way that breaks the
// IdentityServer interface, the build fails immediately with a clear
// error — instead of failing later, mysteriously, when gRPC tries to
// register your service.
var _ csi.IdentityServer = (*Driver)(nil)

Notice NewDriver's signature changed — it now takes a HealthChecker as a third argument. This is a real, deliberate breaking change to code we wrote earlier in the book, and it's worth sitting with why we're doing it here rather than bolting health-checking on some other way: if Probe reached into a global variable, or constructed its own filesystem checker internally, there'd be no way for a test to intercept it. Taking the dependency in through the constructor is what makes it substitutable. TestProbe's own call to NewDriver, written a moment ago, already matches this new three-argument shape — that's the payoff of having written it that way from the start. Two other calls, elsewhere in this file, are not so lucky yet.

go test ./internal/driver/...

./identity_test.go:XX:9: not enough arguments in call to NewDriver
	have (string, string)
	want (string, string, HealthChecker)
./identity_test.go:XX:9: not enough arguments in call to NewDriver
	have (string, string)
	want (string, string, HealthChecker)

These two are TestGetPluginInfo and TestGetPluginCapabilities — written earlier in this chapter, before HealthChecker existed, calling NewDriver with its old two-argument shape. This is the real, visible cost of the breaking change from a moment ago: Go's compiler won't let a stale call site slide by silently just because it used to be valid. Neither test cares about health checking at all — neither one ever calls Probe — so the fix is the smallest one available: pass nil for the third argument in both. HealthChecker is an interface, and a nil interface value is entirely legal in Go; it only becomes a problem if something actually tries to call a method on it, and nothing in either test does.

func TestGetPluginInfo(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0", nil)
	// ...unchanged below this line

func TestGetPluginCapabilities(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0", nil)
	// ...unchanged below this line

Try again:

go test ./internal/driver/...

Every call site now matches, so the whole package builds — and with the build no longer blocked, one more red step surfaces, in exactly the same shape GetPluginInfo's did earlier: the embedded stub answers honestly that Probe isn't implemented yet either.

--- FAIL: TestProbe/backend_is_healthy (0.00s)
    identity_test.go:XX: Probe returned an error: rpc error: code = Unimplemented desc = method Probe not implemented
--- FAIL: TestProbe/backend_is_unhealthy (0.00s)
    identity_test.go:XX: Probe returned an error: rpc error: code = Unimplemented desc = method Probe not implemented
FAIL

Both branches of the table fail the same way, which makes sense — neither one has reached any real logic yet; both are still hitting the same stub. Now Probe itself, in identity.go:

package driver

import (
	"context"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"google.golang.org/protobuf/types/known/wrapperspb"
)

func (d *Driver) Probe(
	ctx context.Context,
	req *csi.ProbeRequest,
) (*csi.ProbeResponse, error) {
	ready := d.health != nil && d.health.Healthy()
	return &csi.ProbeResponse{
		Ready: wrapperspb.Bool(ready),
	}, nil
}

That wrapperspb.Bool(ready) deserves a second look, because the obvious thing to write instead is &ready — a plain *bool — and that produces a confusing compile error the first time you see it: cannot use &ready (value of type *bool) as *wrapperspb.BoolValue value in struct literal. The CSI spec doesn't define ProbeResponse.Ready as a plain boolean; it defines it as a google.protobuf.BoolValue — a tiny wrapper message that holds one boolean field, named Value. This isn't about what a Go *bool can represent — a *bool already distinguishes true, false, and "not set" (nil) just fine. It's about what protobuf itself can represent: proto3 has no native optional scalar type, so a plain bool field can't be told apart from "false" on the wire either. google.protobuf.BoolValue is the standard library's fix for that gap — a message wrapping one field, which protobuf's normal presence rules already handle — and the CSI spec chose it for ProbeResponse.Ready specifically so "not ready" and "hasn't said yet" stay distinguishable across the wire, not just in memory. Go's generated code for the CSI spec follows that wrapper faithfully, so Ready is a *wrapperspb.BoolValue, not a *bool, and building one means either &wrapperspb.BoolValue{Value: ready} or, more idiomatically, the helper constructor wrapperspb.Bool(ready) used above, from google.golang.org/protobuf/types/known/wrapperspb. It's worth noticing that TestProbe's own assertion — resp.GetReady().GetValue() — was already written correctly against this wrapper type: GetValue() only exists because Ready is a message, not a plain bool. The test had the right shape from the start; the implementation just needed to match it.

If this is the first time anything in this project has imported from google.golang.org/protobuf directly, run this once so go.mod records it properly:

go get google.golang.org/protobuf/types/known/wrapperspb

It's likely already sitting in your module graph — the csi package itself depends on it — but Go's module system wants anything you import directly, in your own code, listed as a direct requirement in go.mod, not left as something merely pulled in indirectly through another dependency. Without this, go build can fail with no required module provides package google.golang.org/protobuf/types/known/wrapperspb.

go test ./internal/driver/... -v

--- PASS: TestGetPluginInfo (0.00s)
--- PASS: TestGetPluginCapabilities (0.00s)
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green, including both branches of the table-driven test. This is the payoff, made visible: we just proved Probe correctly reports unhealthy under a condition we can't easily create in real life — a broken storage backend — without ever touching a real disk.

Refactor step: the real health checker

We've been testing against a fake. We still need a real implementation for the actual driver to use once it's running for real. This is a good moment to point at Single Responsibility concretely: this real checker has exactly one job — "can I write to and read from this directory?" — and nothing about Identity, gRPC, or Probe belongs inside it. Create internal/driver/health.go:

package driver

import (
	"os"
	"path/filepath"
)

// LocalDirHealthChecker checks that a given directory exists and is
// writable, by actually writing a small file to it and reading it back.
// It's the "real" implementation of HealthChecker that main.go will use
// — the fakeHealthChecker in identity_test.go exists purely for tests.
type LocalDirHealthChecker struct {
	Root string
}

func (h *LocalDirHealthChecker) Healthy() bool {
	f, err := os.CreateTemp(h.Root, ".health-probe-*")
	if err != nil {
		return false
	}
	path := f.Name()
	defer os.Remove(path)
	defer f.Close()

	if _, err := f.WriteString("ok"); err != nil {
		return false
	}
	_, err = os.ReadFile(path)
	return err == nil
}

var _ HealthChecker = (*LocalDirHealthChecker)(nil)

That last line is the same compile-time interface check pattern from earlier in this chapter, now protecting a second implementation of HealthChecker instead of IdentityServer.

os.CreateTemp matters more than it looks like it should. An earlier version of this checker wrote and read back one fixed name, .health-probe, every time — which works fine for a single caller, but kubelet can and does call Probe from more than one goroutine, and nothing here serializes those calls. Two concurrent probes sharing one fixed filename can race: one probe's os.Remove can delete the file a second probe just wrote, and that second probe's own ReadFile then reports "unhealthy" for a directory that's perfectly fine. CreateTemp gives every call its own uniquely-named file (the * in the pattern gets replaced with a random suffix), so concurrent probes no longer share anything to race over.

This one, unlike everything else in this chapter so far, genuinely does need to touch a real filesystem to test — t.TempDir(), from Go's testing package, gives every test its own directory that's cleaned up automatically:

func TestLocalDirHealthChecker_HealthyRoot(t *testing.T) {
	h := &LocalDirHealthChecker{Root: t.TempDir()}
	if !h.Healthy() {
		t.Error("Healthy() = false, want true for a writable directory")
	}
}

func TestLocalDirHealthChecker_MissingRoot(t *testing.T) {
	h := &LocalDirHealthChecker{Root: filepath.Join(t.TempDir(), "does-not-exist")}
	if h.Healthy() {
		t.Error("Healthy() = true, want false for a root that doesn't exist")
	}
}

func TestLocalDirHealthChecker_UnwritableRoot(t *testing.T) {
	root := t.TempDir()
	if err := os.Chmod(root, 0o500); err != nil {
		t.Fatalf("os.Chmod: %v", err)
	}
	t.Cleanup(func() { os.Chmod(root, 0o750) })

	h := &LocalDirHealthChecker{Root: root}
	if h.Healthy() {
		t.Error("Healthy() = true, want false for a read-only directory")
	}
}

Three real filesystem states, three real outcomes: a writable directory, a root that was never created, and a root that exists but refuses writes. Probe's own branching logic was already fully covered by the fake HealthChecker earlier in this chapter — these three tests cover the one thing the fake couldn't: whether LocalDirHealthChecker itself correctly turns "can I write and read a file here" into true or false against a real directory.

Add a test target to the Makefile

Same reasoning as make deploy in Chapter 2 — you'll run this constantly:

test:
	go test ./... -v

From here on, "run the tests" means make test.

What just happened, and why it mattered

We wrote three methods, and every line of production code exists because a failing test demanded it, not because we typed it and hoped. Along the way, one real design decision came out of that process rather than being imposed on it from outside: Probe doesn't know or care how health gets checked, because depending on an interface instead of a concrete filesystem call is what let us test the interesting logic (the branching in Probe) completely separately from the boring, hard-to-test logic (actually touching a disk).

That split — interesting branching logic tested with fakes, boring I/O tested separately or not at all yet — is a pattern you'll see repeat in every chapter from here on, especially once Controller and Node need to call out to real storage operations.

Wiring up the gRPC server

Every method we built above only exists as Go code that tests call directly — nothing's listening on the network yet. Create cmd/localdir-csi/main.go:

package main

import (
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
	socketPath    = "/csi/csi.sock"
)

func main() {
	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	// This is the one line from Chapter 2.5 that's different from a
	// typical gRPC tutorial: "unix" instead of "tcp", and a file path
	// instead of a port number.
	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: "/data"},
	)

	server := grpc.NewServer()

	// This line connects our Driver struct to gRPC's Identity service
	// machinery. From this point on, whenever a GetPluginInfo /
	// GetPluginCapabilities / Probe call arrives on the socket, gRPC
	// routes it to the matching method on d.
	csi.RegisterIdentityServer(server, d)

	// Reflection is a second, separate gRPC service, distinct from
	// Identity, that answers one question: "what services and methods do
	// you have, and what do their messages look like?" A real CSI sidecar
	// never needs to ask that question — it's compiled against the exact
	// same generated csi package we are, so it already knows. A tool like
	// grpcurl, run by a human from a terminal, has no such advance
	// knowledge, and no .proto file handy either, unless it's told where
	// to look. Reflection is how it asks the server directly instead.
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s", socketPath)

	// Serve blocks forever, handling calls as they arrive, until
	// something goes wrong or the process is killed.
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

Nothing here is CSI-specific except the socket path and which Register*Server function we call. This is the same shape every gRPC server in Go has: get a listener, make a server, register your service(s) against it, call Serve.

Notice main.go builds the real LocalDirHealthChecker, pointed at /data, and hands it to NewDriver. Nothing in main.go needs to know HealthChecker is an interface with a test-only fake implementation elsewhere — it just needs something that satisfies Healthy() bool, and that's exactly the point of building it this way.

Also notice we only called csi.RegisterIdentityServer — not Controller or Node. That's deliberate. Right now Driver doesn't implement csi.ControllerServer or csi.NodeServer (we haven't written those methods yet), so registering it as those services would fail. We'll add those registrations — and the matching UnimplementedControllerServer / UnimplementedNodeServer embeds in driver.go — in Chapters 5 and 7 as we implement each service.

reflection.Register(server), on the other hand, only needs to be called once, no matter how many CSI services Driver ends up implementing — it inspects whatever's already registered on server and answers questions about all of it. Nothing about this line is CSI-specific either; it's the same one-liner in any Go gRPC server that wants to be inspectable by grpcurl, Postman, or similar tools.

Running it

Since the socket path is /csi/csi.sock, and we're not inside a container yet, we need somewhere in your project directory to stand in for both the socket and the /data directory the health checker writes to. Add a run target to the Makefile from Chapter 2:

run:
	mkdir -p ./csi ./data
	go run ./cmd/localdir-csi

And temporarily change socketPath in main.go to ./csi/csi.sock, and the health checker's Root to ./data, for this local test — we'll switch both back to their container paths (/csi/csi.sock and /data) in Chapter 4 when we deploy for real; inside a pod, those paths won't collide with anything on your laptop.

In one terminal:

make run

You should see:

localdir-csi listening on ./csi/csi.sock

It's just sitting there, waiting. That's correct — a server with nothing to do until someone calls it.

Talking to it

We need a way to make a gRPC call from a terminal, the same way curl lets you make an HTTP call. The tool for that is grpcurl:

go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest

In a second terminal, ask it who it is:

grpcurl -plaintext unix:./csi/csi.sock csi.v1.Identity/GetPluginInfo

The unix: prefix on the target — not a separate -unix flag — is what tells grpcurl (and, underneath it, gRPC's own connection logic) to treat ./csi/csi.sock as a filesystem path rather than a host:port pair. unix: followed directly by a path, with no slashes in between, accepts either a relative or an absolute path exactly as written. If you ever connect to an absolute socket path instead, the equivalent form is three slashes — unix:///absolute/path/to/csi.sock — because the third slash is actually the first character of the path itself.

You should get back:

{
  "name": "localdir.csi.example.com",
  "vendorVersion": "0.1.0"
}

That JSON is grpcurl displaying the GetPluginInfoResponse your method returned — translated from Protobuf binary back into something readable, purely for your benefit as a human. The real caller (a sidecar) would get the actual typed Go struct, not JSON text.

Try the other two:

grpcurl -plaintext unix:./csi/csi.sock csi.v1.Identity/GetPluginCapabilities
grpcurl -plaintext unix:./csi/csi.sock csi.v1.Identity/Probe

GetPluginCapabilities should echo back the CONTROLLER_SERVICE capability you wrote. Probe should return {"ready": true} — because ./data exists (you created it with make run) and LocalDirHealthChecker was able to write and read a file in it. If you delete ./data while the server's running and call Probe again, you should now see {"ready": false} — a real, live demonstration of the exact branch your TestProbe table already covers, now visible over the network instead of just in a test.

If any of these hang instead of returning immediately, double check the socket path matches on both sides — a mistyped path is the single most common mistake at this stage, and it fails silently (grpcurl just can't connect) rather than with an obvious error. A different mistake produces a loud, immediate error instead of a hang: leaving off the unix: prefix — or restoring an old -unix flag from a tutorial you half-remember — makes grpcurl treat the socket path as a host:port address, and it fails right away with something like dial tcp: address ./csi/csi.sock: missing port in address. That error means the target scheme was wrong, not that the server is unreachable.

What just happened, one more time, plainly

You wrote a Go struct with a dependency it doesn't control the implementation of. You told gRPC "this struct answers Identity questions." You started listening on a file (not a network port). A separate tool sent a request to that file, gRPC turned the bytes into a Go function call, your function ran — checking a real directory through an interface your tests had already exercised with a fake — and gRPC turned the return value back into bytes for grpcurl to display.

That entire round trip — socket, gRPC routing, your method, back again — is identical for every RPC you'll write for the rest of this book. Only the method names and what they do will change. Controller's CreateVolume and Node's NodePublishVolume arrive through this exact same pipe, and we'll write both test-first, the same way, starting in Chapter 5.

What you should have now

• internal/driver/driver.go with a Driver struct — embedding csi.UnimplementedIdentityServer for forward compatibility — a HealthChecker interface, a constructor that takes one in, and a compile-time check (var _ csi.IdentityServer = (*Driver)(nil)) confirming Driver satisfies csi.IdentityServer
• internal/driver/identity.go implementing all three Identity methods, each one overriding the embedded stub of the same name
• internal/driver/health.go with LocalDirHealthChecker, the real implementation used outside tests
• internal/driver/identity_test.go with passing tests for all three methods, written before their implementations, using t.Context() rather than context.Background()
• cmd/localdir-csi/main.go starting a gRPC server on a Unix socket, with reflection enabled so grpcurl can inspect it
• A make test target, and a running driver you've personally called over gRPC and gotten real answers from — including watching Probe flip from ready to not-ready in real time

Next chapter, we stop running this by hand and get it into the kind cluster for real — building the container image, deploying it as a DaemonSet alongside the node-driver-registrar sidecar, and watching kubelet discover it for the first time.

Chapter 4: Registering with Kubelet

Right now, localdir-csi is a binary you run by hand, on your own laptop, listening on a socket in a directory you created yourself. That's been exactly the right amount of complexity for learning how the Identity service works. It is not, however, a driver Kubernetes knows exists. Nothing about kind create cluster or kubectl has any idea your binary is out there.

This chapter closes that gap: your driver goes into a container image, that image gets deployed to your kind cluster as a real Kubernetes workload, and — with the help of one small sidecar — kubelet finds out it exists. By the end, you'll have watched a real Kubernetes node discover your driver over a socket, the same way it discovers every CSI driver in every real cluster. You'll also hit a genuine wall, on purpose: kubelet wants one more thing from your driver before it'll fully trust it, and we haven't written that thing yet. That wall is exactly what Chapter 5 is for.

One binary, two homes

Open cmd/localdir-csi/main.go as it stands at the end of Chapter 3. Two values are hardcoded as constants: the socket path (/csi/csi.sock) and the health checker's root directory (/data). Those are the right paths for a container — they're where we'll mount things in a moment — but they're the wrong paths to run directly on your laptop, where writing to /csi and /data means writing to your machine's real filesystem root, which will either fail outright (no permission) or, worse, succeed and leave files somewhere you didn't intend.

Chapter 3 sidestepped this by having you hardcode local-friendly values — ./csi/csi.sock and ./data — instead, with a note that we'd deal with the container paths "for real" once we actually had a container to put them in. Now we do, and the honest problem is: one binary needs to listen in two different places, depending on where it's running, and a hardcoded constant can only ever be right in one of them.

This is what command-line flags are for. You've been passing flags to other programs all through this book — kind create cluster --name csi-dev — without needing to write one yourself yet. Go's standard library ships everything for it in the flag package: you declare a named flag, a default value, and a one-line description; call flag.Parse() once, early in main; and from then on you have a value that's whatever the program was actually invoked with, or the default if it wasn't given.

We'll add two: -endpoint, for where the driver listens, and -data-dir, for where the health checker looks. Defaults matter here — default to whatever's right for the container, since that's where this binary spends most of its life, and override explicitly for local runs.

A small, testable piece: turning an endpoint into a socket path

-endpoint needs a default value, and it's worth choosing that value deliberately rather than arbitrarily. Real CSI drivers — the ones this book is modeled on — universally accept their endpoint as a URI with a unix:// scheme, not a bare path: something like unix:///csi/csi.sock. You've already met this exact scheme once, in Chapter 3, typing it yourself on the grpcurl command line. Reusing it here isn't a coincidence — it's the same idea ("this is a filesystem path, not a host:port pair") showing up on the server side instead of the client side, and it means anyone who's used grpcurl against a CSI driver before will recognize the convention immediately.

The catch, also from Chapter 3: unix:// URIs are easy to get subtly wrong. unix:///csi/csi.sock (three slashes) is correct for an absolute path. unix:./csi/csi.sock (one colon, no slashes at all) is correct for a relative one. unix://./csi/csi.sock (two slashes plus a relative path) looks plausible and is silently wrong — it hands . to the URI's host portion and drops the leading . from the path entirely. Since net.Listen has no idea what a unix:// URI is — it wants a plain filesystem path — something in our code has to parse the endpoint string and pull the real path back out, correctly, for all of these forms.

That parsing is a small, pure function: string in, path out, no network, no gRPC, nothing that needs a live server to test. Exactly the kind of thing to write test-first.

Create internal/driver/endpoint_test.go:

package driver

import "testing"

func TestSocketPathFromEndpoint(t *testing.T) {
	tests := []struct {
		name     string
		endpoint string
		want     string
		wantErr  bool
	}{
		{name: "absolute path, three slashes", endpoint: "unix:///csi/csi.sock", want: "/csi/csi.sock"},
		{name: "relative path, one colon, no slashes", endpoint: "unix:./csi/csi.sock", want: "./csi/csi.sock"},
		{name: "wrong scheme is rejected", endpoint: "tcp://localhost:8080", wantErr: true},
		{name: "missing scheme is rejected", endpoint: "/csi/csi.sock", wantErr: true},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			got, err := SocketPathFromEndpoint(tt.endpoint)
			if tt.wantErr {
				if err == nil {
					t.Fatalf("SocketPathFromEndpoint(%q) = %q, want an error", tt.endpoint, got)
				}
				return
			}
			if err != nil {
				t.Fatalf("SocketPathFromEndpoint(%q) returned an error: %v", tt.endpoint, err)
			}
			if got != tt.want {
				t.Errorf("SocketPathFromEndpoint(%q) = %q, want %q", tt.endpoint, got, tt.want)
			}
		})
	}
}

Run it:

go test ./internal/driver/...

Red, the way every first test in this book has been red, because SocketPathFromEndpoint doesn't exist yet:

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
./endpoint_test.go:20:16: undefined: SocketPathFromEndpoint
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]
FAIL

Now write it. Per Single Responsibility — the same principle Chapter 3 pointed at for LocalDirHealthChecker — this doesn't belong crammed into driver.go alongside the Driver struct itself; parsing an endpoint string is a self-contained job with nothing to do with what Driver is or does. Give it its own file, internal/driver/endpoint.go:

package driver

import (
	"fmt"
	"net/url"
)

// SocketPathFromEndpoint extracts a filesystem path from a "unix:"
// endpoint URI. This is deliberately the same parsing rule grpc-go's own
// unix resolver uses internally — url.Path for a "unix://" form with the
// double slash ("unix:///abs/path"), falling back to url.Opaque for the
// single-colon form without one ("unix:relative/path") — so a value that
// behaves correctly as a grpcurl target also behaves correctly here.
func SocketPathFromEndpoint(endpoint string) (string, error) {
	u, err := url.Parse(endpoint)
	if err != nil {
		return "", fmt.Errorf("parsing endpoint %q: %w", endpoint, err)
	}
	if u.Scheme != "unix" {
		return "", fmt.Errorf("endpoint %q has scheme %q, want \"unix\"", endpoint, u.Scheme)
	}
	path := u.Path
	if path == "" {
		path = u.Opaque
	}
	if path == "" {
		return "", fmt.Errorf("endpoint %q has no socket path", endpoint)
	}
	return path, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green — all four cases, including both rejected forms.

Updating main.go to use it

With SocketPathFromEndpoint written and tested, wiring it into main.go is mechanical. Replace the whole file:

package main

import (
	"flag"
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
)

func main() {
	endpoint := flag.String("endpoint", "unix:///csi/csi.sock",
		"gRPC endpoint the driver listens on, as a unix:// URI")
	dataDir := flag.String("data-dir", "/data",
		"directory the health check reads and writes to confirm local storage is usable")
	flag.Parse()

	socketPath, err := driver.SocketPathFromEndpoint(*endpoint)
	if err != nil {
		log.Fatalf("invalid -endpoint: %v", err)
	}

	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: *dataDir},
	)

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s", socketPath)
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

Everything below the flag declarations is exactly what Chapter 3 built — the only change is that socketPath and *dataDir are now read from the command line instead of baked in.

Update the run target in your Makefile to pass local-friendly values explicitly, since the defaults are now the container's paths, not your laptop's:

run:
	mkdir -p ./csi ./data
	go run ./cmd/localdir-csi -endpoint=unix:./csi/csi.sock -data-dir=./data

make run and every grpcurl command from Chapter 3 still work exactly as before — the only thing that changed is that they now say so explicitly on the command line instead of it being the only option.

Packaging it: the Dockerfile

Kubernetes doesn't run Go binaries. It runs containers — a container image is a filesystem plus some metadata (what to run, as what user) bundled up so it can be shipped to any machine and run identically there. A Dockerfile is the recipe for building one: a short script of instructions, each producing one layer of that filesystem.

We want two very different things out of this build. First, a Go toolchain, to compile cmd/localdir-csi into a binary — that toolchain is enormous (hundreds of megabytes) and completely useless once the binary exists. Second, the smallest possible image to actually ship and run — every extra file in a running container is one more thing that could theoretically be exploited if the container is ever compromised. A multi-stage build is Docker's answer to wanting both: a Dockerfile with more than one FROM line, where each FROM starts a fresh, separate filesystem, and a later stage can selectively copy specific files out of an earlier one — the compiled binary, and nothing else the compiler needed to produce it.

Each instruction below does one plain thing: FROM picks the starting filesystem for a stage, WORKDIR picks the directory later instructions run in, COPY copies files in from outside the image, RUN runs a command while the image is being built (not later, when it's running), and ENTRYPOINT is the one command that actually runs when a container starts from the finished image.

Create Dockerfile in the project root:

# Stage 1: compile the binary. This image has the full Go toolchain and
# nothing about it ends up in the final image except whatever we
# explicitly COPY out of it below.
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/localdir-csi ./cmd/localdir-csi

# Stage 2: the image that actually ships. "distroless" means exactly what
# it sounds like — no shell, no package manager, no coreutils, nothing
# beyond the C library our binary is linked against. There's nothing here
# for an attacker to get a shell with even if they found a way in, and
# nothing here to patch for vulnerabilities that don't apply to a program
# that never asked for them.
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/localdir-csi /localdir-csi
ENTRYPOINT ["/localdir-csi"]

CGO_ENABLED=0 matters specifically because of that second stage: distroless/static has no C library at all, so a binary that dynamically links against one (the default, if any dependency ever pulls in cgo) would fail to start with a missing-library error the moment it hit that image. Setting CGO_ENABLED=0 tells Go's compiler to produce a fully self-contained binary instead — nothing to link against at runtime, because nothing was linked in the first place.

make build from Chapter 2 already runs docker build -t localdir-csi:dev . — it just had nothing to find before now. Try it:

make build

What node-driver-registrar is actually for

Before looking at where its socket lives or what it logs, it's worth being clear about the one job node-driver-registrar exists to do, in plain terms: it's the thing that tells kubelet your driver exists at all.

Think of kubelet as a building's front-desk security guard, deciding which companies are allowed to serve people on that floor. Your driver is a brand-new company that just moved in — it's up and running, fully capable of doing its job, but nobody at the front desk has heard of it yet. It's just a process quietly listening on a socket file, and a socket file sitting on disk isn't something kubelet automatically notices.

node-driver-registrar's entire purpose is to walk up to that front desk on your driver's behalf and say "there's a new company here, this is its name, and this is the door it's behind." Concretely, it does that in two steps: first it asks your own driver "what's your name?" (a gRPC call to GetPluginInfo, the exact method Chapter 3 wrote), and then it writes that name, plus the path to your driver's socket, into a directory kubelet is constantly watching. The moment that new information shows up, kubelet notices it and goes to introduce itself to your driver directly.

Without node-driver-registrar, none of that introduction happens. localdir-csi could be running perfectly, listening on its socket, completely healthy — and kubelet would simply never know to knock on its door, the same way a security guard has no reason to check a floor for new tenants nobody told them about.

Where the driver actually lives on a node

Chapter 1 described sidecars as sharing a Unix socket with your driver over an emptyDir volume — true, and it's exactly how the Controller side will work starting in Chapter 7, where external-provisioner and your Controller service are two containers in the very same pod. The Node side, and specifically node-driver-registrar, needs one more layer of precision on that idea, because of who has to reach the socket.

An emptyDir volume is only visible to containers inside the same pod. Kubelet is not a container in your pod — it's a process running directly on the node itself, outside of Kubernetes entirely, the same way dockerd or sshd might be. For kubelet to ever open your driver's socket, that socket has to live somewhere kubelet can already see without going through Kubernetes at all: a real directory on the node's own filesystem. That's a hostPath volume — instead of Kubernetes conjuring up empty, pod-scoped storage, it hands your container a bind mount straight into a path that already exists on the host.

By convention, every CSI driver's socket lives at /var/lib/kubelet/plugins/<driver-name>/ on the host — kubelet expects one subdirectory per driver, named after that driver's own name. Mount that same host directory into both your driver container and the registrar sidecar, at whatever path each one is configured to look for it, and both containers — and kubelet itself, reading straight off the host — are looking at the exact same file.

The DaemonSet

Chapter 1 already told you why the Node service runs as a DaemonSet rather than an ordinary Deployment: it needs to run on every node, because "attached to a node" and "mounted into a pod on that node" are both inherently per-node facts, not cluster-wide ones. A DaemonSet is Kubernetes's object for exactly that shape — one pod per node, automatically, including on any node added to the cluster later.

Create deploy/node.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: localdir-csi-node
  labels:
    app: localdir-csi
spec:
  selector:
    matchLabels:
      app: localdir-csi
  template:
    metadata:
      labels:
        app: localdir-csi
    spec:
      containers:
        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/localdir.csi.example.com/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: data-dir
          hostPath:
            path: /var/lib/localdir-csi/data
            type: DirectoryOrCreate

A few things worth reading carefully here, since they're easy to get subtly wrong:

imagePullPolicy: IfNotPresent matters specifically because of how you're getting this image onto the cluster at all. There's no registry involved — make load (which make deploy depends on, from Chapter 2) runs kind load docker-image, which copies your locally built image directly into every kind node. Without IfNotPresent, kubelet's default behavior would be to try pulling localdir-csi:dev from a real registry, fail, and never discover the image already sitting right there on the node.

socket-dir is mounted at /csi in both containers, backed by the exact same hostPath. Your driver listens at /csi/csi.sock inside its own container (that's the default we just wired into -endpoint); the registrar's --csi-address=/csi/csi.sock points at the same path inside its container — same file, reached through two different containers' mount points, because both mount points resolve back to the same directory on the host.

--kubelet-registration-path, by contrast, is not a path inside any container — it's the path kubelet itself, running on the host, will use to reach that socket. That's why it's the host path (/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock), matching socket-dir's hostPath.path plus the socket filename, rather than /csi/csi.sock. Mixing these two up — giving the registrar its own container path where kubelet expects a host path — is a common mistake, and the failure mode is unhelpful: kubelet will simply never find a working socket at the path it was told, with no error pointing at the actual cause.

registration-dir uses type: Directory, not DirectoryOrCreate, on purpose: /var/lib/kubelet/plugins_registry/ is a directory kubelet itself manages and already creates on every node it runs on. If it's missing, something more fundamental than your driver's manifest is wrong, and creating it out from under kubelet isn't the right fix. socket-dir and data-dir, on the other hand, are specific to this driver and won't exist until something creates them — DirectoryOrCreate lets Kubernetes do that the first time this DaemonSet runs.

data-dir isn't used by anything in this chapter — nothing calls Probe against the deployed driver yet — but it's the same /data path LocalDirHealthChecker has depended on since Chapter 3, now given somewhere real on the host to actually write to, instead of silently failing inside an empty container filesystem.

Telling Kubernetes about the driver, cluster-wide: the CSIDriver object

The DaemonSet gets your driver's code running on every node. Kubernetes also wants a small, separate, cluster-scoped object describing facts about the driver itself — not "where is it running," but "what does it need from the rest of the system." That object is CSIDriver, and unlike the DaemonSet, there's exactly one of these per driver, cluster-wide, not one per node.

Create deploy/csidriver.yaml:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: localdir.csi.example.com
spec:
  attachRequired: false
  podInfoOnMount: false
  volumeLifecycleModes:
    - Persistent

attachRequired is the one to get right immediately, and it's worth understanding why rather than just copying the value. When true (the default, if this field is left out entirely), Kubernetes will wait for a separate ControllerPublishVolume call — "attach this volume to this node" — to succeed before it ever tries to mount that volume into a pod. That's the right behavior for storage that genuinely has an attach step, like an EBS volume being attached to an EC2 instance. localdir-csi never will — Chapter 8 implements ControllerPublishVolume for the sake of completeness and to show the pattern, but our storage is already local to every node by definition, so there's nothing to attach. Setting attachRequired: false now tells Kubernetes not to wait for a step this driver — as of this chapter, and permanently in spirit — doesn't need.

podInfoOnMount: false (also the field's default) means Kubernetes won't bother passing pod metadata — name, namespace, service account — into NodePublishVolume calls. Some drivers need that context to make mount-time decisions; ours doesn't, so there's no reason to ask for it.

volumeLifecycleModes: [Persistent] states the obvious but required fact that this driver's volumes are the ordinary kind — created ahead of time via a PersistentVolumeClaim, existing independently of any one pod — as opposed to Ephemeral inline volumes, which this book doesn't cover.

Deploying

Everything's in place. Build, load, and apply:

make deploy

Watch the pods come up:

kubectl get pods -l app=localdir-csi -w

You should see one pod per node in your cluster (just one, for the single-node csi-dev cluster from Chapter 2), with 2/2 containers — localdir-csi and node-driver-registrar — reported as Running.

What actually happens when kubelet finds your socket

Registration isn't one gRPC call — it's a small handshake, and it's worth walking through concretely, because you're about to watch part of it fail, honestly, for a reason that makes complete sense once you see it.

node-driver-registrar's job splits into two halves. First, it acts as a gRPC client to your driver: it dials --csi-address and calls GetPluginInfo, the exact method you wrote in Chapter 3, purely to learn your driver's name. Check its logs to see this happen for real:

kubectl logs -l app=localdir-csi -c node-driver-registrar --tail=50

Somewhere in there you should find lines close to:

Attempting to open a gRPC connection  csiAddress="/csi/csi.sock"
Calling CSI driver to discover driver name
CSI driver name  csiDriverName="localdir.csi.example.com"

Second, having learned its own driver's name, the registrar becomes a gRPC server itself — a tiny one, implementing a completely different, Kubernetes-internal service (not CSI at all) whose only job is answering "what plugin are you, and where's its socket?" It listens for that on its own socket, inside registration-dir. Kubelet's plugin watcher polls that directory continuously; the moment a new socket shows up there, it connects and asks exactly that question.

Getting an answer to that question is where kubelet's own work begins, separately from anything node-driver-registrar does. Kubelet dials your driver's socket directly — the real, actual /var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock, the same file, not through the registrar at all — and, among other calls, tries to invoke NodeGetInfo. That's a method on csi.NodeServer. We haven't written a Node service yet. Not one method of it exists, and main.go never even calls csi.RegisterNodeServer — the whole service is simply absent from what your driver exposes.

You can watch kubelet discover exactly that, from your own laptop, without touching the cluster at all — it's the same failure, over the same kind of socket, that Chapter 3 already gave you the tools to reproduce:

make run

and, in a second terminal:

grpcurl -plaintext unix:./csi/csi.sock csi.v1.Node/NodeGetInfo

ERROR:
  Code: Unimplemented
  Message: unknown service csi.v1.Node

That's not "not implemented" in the polite, UnimplementedNodeServer sense Chapter 3 walked through for Identity — there's no csi.v1.Node service registered on this server at all, so gRPC can't even find a stub to answer with. It's the plainest, earliest kind of "this doesn't exist yet" a gRPC server can report.

Back in the cluster, kubelet gets that exact same error when it tries NodeGetInfo for real, and it treats that as a failed registration — which node-driver-registrar takes seriously enough to exit and let Kubernetes restart it, rather than silently limping along half-registered. Watch it happen:

kubectl get pods -l app=localdir-csi -w

Give it a few seconds and you should see the RESTARTS count on the pod start climbing. Because the container that just crashed is gone by the time you'd normally check its logs, you need --previous to see what it said on its way out:

kubectl logs -l app=localdir-csi -c node-driver-registrar --previous

Look for a line mentioning registration failing — the exact wording varies by version, but the cause underneath it is always this same missing NodeGetInfo.

One more angle on the same fact, using an object you haven't met yet. Kubernetes tracks which CSI drivers are actually, successfully registered on each node in a CSINode object — one per node, cluster-scoped, not something your driver creates or touches directly; kubelet manages it entirely on its own, updating it only once a driver's registration fully succeeds.

kubectl get nodes
kubectl get csinode <node-name> -o yaml

Look under spec.drivers. localdir.csi.example.com isn't there. As far as Kubernetes is concerned, your driver introduced itself, and kubelet still doesn't trust it as a working node plugin — because it genuinely isn't one yet.

That's the honest, unglamorous state to end this chapter on: a driver that's discoverable, but not yet complete. NodeGetInfo is the very first method of the Node service, and it's next.

What you should have now

• internal/driver/endpoint.go with a tested SocketPathFromEndpoint helper, parsing unix: endpoint URIs the same way grpc-go's own resolver does
• cmd/localdir-csi/main.go reading its socket path and health-check directory from -endpoint and -data-dir flags, defaulting to the container's paths
• A Dockerfile building a small, distroless image via a multi-stage build
• deploy/node.yaml: a DaemonSet running your driver alongside node-driver-registrar, sharing a hostPath-backed socket directory
• deploy/csidriver.yaml: a CSIDriver object declaring attachRequired: false and volumeLifecycleModes: [Persistent]
• A driver pod running in your kind cluster, its identity confirmed by node-driver-registrar's logs, and node-driver-registrar itself crash-looping for a reason you've reproduced locally and fully understand
• Direct, hands-on proof — via csi.v1.Node/NodeGetInfo and kubectl get csinode — of exactly what kubelet still needs from you

Chapter 5 writes NodeGetInfo and NodeGetCapabilities: the first two methods of the Node service, test-first, the same way every method in Chapter 3 was written. Once NodeGetInfo exists, the exact same DaemonSet you just wrote, completely unchanged, will register successfully — you'll rerun the same kubectl get csinode command from this chapter and watch the answer change.

Chapter 5: The Node Service, Part 1

Chapter 4 ended on an unresolved problem, on purpose: kubelet tries to call NodeGetInfo on your driver as the last step of registering it, that method doesn't exist yet, and node-driver-registrar crash-loops because of it. This chapter writes NodeGetInfo, and its neighbor NodeGetCapabilities, the same test-first way every method in this book gets written. By the end, the exact DaemonSet you already deployed — unchanged — will register successfully.

What these two methods are for

Identity answers questions about the driver as a whole: "who are you," "what can you do." The Node service answers the same two questions, but scoped to one specific node — the one this particular copy of your driver happens to be running on, since a DaemonSet runs one pod per node and each pod only ever knows about its own machine.

• NodeGetInfo — "who are you, as this node?" It returns an identifier for this node — something the rest of Kubernetes can use later to say "attach this volume to node X" — plus, optionally, a cap on how many volumes this node can hold and where it sits topologically (which region, zone, or rack). This is the exact method kubelet calls, once, as the last step of registration.
• NodeGetCapabilities — "what are you able to do, on this node?" Some Node plugins support a separate "staging" step before mounting, or can report live volume usage statistics, or support expanding a volume without unmounting it first. Ours doesn't do any of that yet, so — honestly, for now — the answer is "nothing beyond the required basics."

Both, like every Identity method in Chapter 3, are plain request/response calls: no filesystem access, no branching on real conditions. That keeps this chapter's actual new territory narrow and lets it focus on something more interesting than the RPCs themselves — where a node's identity should actually come from.

Driver gains a second embedded interface

Node is a separate gRPC service from Identity, with its own Unimplemented* stub, for the exact reason Chapter 3 walked through for csi.IdentityServer: the real csi.NodeServer interface requires an unexported mustEmbedUnimplementedNodeServer() method that only embedding csi.UnimplementedNodeServer can satisfy, so that a future CSI spec version can add a new Node method without breaking every driver that predates it.

Update internal/driver/driver.go:

package driver

import (
	"github.com/container-storage-interface/spec/lib/go/csi"
)

// HealthChecker describes anything that can report whether the driver's
// backend is usable right now. Identity's Probe method depends on this
// interface — not on any specific way of checking health — so that a
// real filesystem check and a test fake are equally valid things to hand
// it. This is the whole idea behind Dependency Inversion: Driver names
// what it needs, not how that need gets satisfied.
type HealthChecker interface {
	Healthy() bool
}

// Driver holds everything our CSI driver needs to answer gRPC calls.
type Driver struct {
	csi.UnimplementedIdentityServer
	csi.UnimplementedNodeServer

	name    string
	version string
	health  HealthChecker
	nodeID  string
}

// NewDriver builds a Driver. health can be nil — we guard against that in
// Probe — which is useful for tests that don't care about health checks,
// but main.go always passes a real implementation.
func NewDriver(name, version string, health HealthChecker, nodeID string) *Driver {
	return &Driver{
		name:    name,
		version: version,
		health:  health,
		nodeID:  nodeID,
	}
}

// Compile-time checks: these lines do nothing at runtime. They exist
// purely so that if you ever change Driver in a way that breaks either
// interface, the build fails immediately with a clear error — instead of
// failing later, mysteriously, when gRPC tries to register your service.
var _ csi.IdentityServer = (*Driver)(nil)
var _ csi.NodeServer = (*Driver)(nil)

Embedding two different Unimplemented* structs in the same struct is completely ordinary Go — Driver now has two sets of promoted stub methods, one per service, and since IdentityServer and NodeServer don't share any method names, there's no conflict between them. Each compile-time check only cares about its own interface, so both can — and should — live here side by side, exactly the way IdentityServer's did on its own in Chapter 3.

NewDriver also grew a fourth parameter, nodeID string — and this breaks something. Run the tests:

go test ./internal/driver/...

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
internal/driver/identity_test.go:10:54: not enough arguments in call to NewDriver
	have (string, string, nil)
	want (string, string, HealthChecker, string)
internal/driver/identity_test.go:26:54: not enough arguments in call to NewDriver
	have (string, string, nil)
	want (string, string, HealthChecker, string)
internal/driver/identity_test.go:66:56: not enough arguments in call to NewDriver
	have (string, string, *fakeHealthChecker)
	want (string, string, HealthChecker, string)
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]
FAIL

The same lesson Chapter 3 taught the first time NewDriver's signature changed: every existing call site breaks at once, and every one has to be updated, not just the one driving the change. None of these three tests care what node they're pretending to run on, so the fix is the smallest one available — a placeholder string, the same way nil was already the placeholder for a health checker these tests don't need either. In identity_test.go, update all three:

d := NewDriver("localdir.csi.example.com", "0.1.0", nil, "test-node")

d := NewDriver("localdir.csi.example.com", "0.1.0", nil, "test-node")

d := NewDriver("localdir.csi.example.com", "0.1.0", &fakeHealthChecker{healthy: tt.healthy}, "test-node")

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.012s

Green again, and worth noticing what didn't happen here: nodeID became a plain constructor parameter, not a second interface like HealthChecker. That's deliberate, not an inconsistency. Dependency Inversion earns its keep in Probe because the test needs to control behavior — force the healthy branch, force the unhealthy branch, and watch Probe react differently to each. NodeGetInfo never branches on its node ID at all; it just hands back whatever value it was given. When there's no behavior to substitute, wrapping a value in an interface adds a layer of indirection with nothing behind it to justify it — a plain parameter says exactly as much as the code needs to say, no more.

Writing NodeGetInfo, test-first

One method at a time, the same rhythm every RPC in this book follows: write one test, watch it fail, write only enough code to pass it, then move to the next one. Create internal/driver/node_test.go:

package driver

import (
	"testing"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func TestNodeGetInfo(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0", nil, "test-node-1")

	resp, err := d.NodeGetInfo(t.Context(), &csi.NodeGetInfoRequest{})
	if err != nil {
		t.Fatalf("NodeGetInfo returned an error: %v", err)
	}

	if resp.NodeId != "test-node-1" {
		t.Errorf("NodeId = %q, want %q", resp.NodeId, "test-node-1")
	}
}

Run it:

go test ./internal/driver/... -v

Red, in the exact shape Chapter 3 taught you to expect once a service's Unimplemented* stub is embedded from the start: not a compile error — NodeGetInfo already exists, promoted from UnimplementedNodeServer — but a real, honest runtime answer that it isn't implemented yet.

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
    node_test.go:14: NodeGetInfo returned an error: rpc error: code = Unimplemented desc = method NodeGetInfo not implemented
--- FAIL: TestNodeGetInfo (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Chapter 3's tests still pass — nothing about them changed — and the one new test fails exactly the way a freshly-embedded, not-yet-implemented method should. Now write just enough to fix it, in a new file — node.go, parallel to identity.go, same as the project layout Chapter 2 sketched out before either file existed:

package driver

import (
	"context"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func (d *Driver) NodeGetInfo(
	ctx context.Context,
	req *csi.NodeGetInfoRequest,
) (*csi.NodeGetInfoResponse, error) {
	return &csi.NodeGetInfoResponse{
		NodeId: d.nodeID,
	}, nil
}

go test ./internal/driver/... -v

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
--- PASS: TestNodeGetInfo (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green.

Writing NodeGetCapabilities, test-first

Same rhythm, next method. Add this test to internal/driver/node_test.go, below TestNodeGetInfo:

func TestNodeGetCapabilities(t *testing.T) {
	d := NewDriver("localdir.csi.example.com", "0.1.0", nil, "test-node-1")

	resp, err := d.NodeGetCapabilities(t.Context(), &csi.NodeGetCapabilitiesRequest{})
	if err != nil {
		t.Fatalf("NodeGetCapabilities returned an error: %v", err)
	}

	if len(resp.Capabilities) != 0 {
		t.Errorf("got %d capabilities, want 0", len(resp.Capabilities))
	}
}

go test ./internal/driver/... -v

Red, and only for the method that's actually new — TestNodeGetInfo, written and satisfied a moment ago, keeps passing:

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
--- PASS: TestNodeGetInfo (0.00s)
=== RUN   TestNodeGetCapabilities
    node_test.go:27: NodeGetCapabilities returned an error: rpc error: code = Unimplemented desc = method NodeGetCapabilities not implemented
--- FAIL: TestNodeGetCapabilities (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Add NodeGetCapabilities to node.go, below NodeGetInfo:

func (d *Driver) NodeGetCapabilities(
	ctx context.Context,
	req *csi.NodeGetCapabilitiesRequest,
) (*csi.NodeGetCapabilitiesResponse, error) {
	return &csi.NodeGetCapabilitiesResponse{
		Capabilities: []*csi.NodeServiceCapability{},
	}, nil
}

go test ./internal/driver/... -v

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
--- PASS: TestNodeGetInfo (0.00s)
=== RUN   TestNodeGetCapabilities
--- PASS: TestNodeGetCapabilities (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green. NodeGetCapabilities returning an empty slice isn't a shortcut — it's the honest answer for a driver that doesn't yet implement staging, volume statistics, or expansion. The same Open/Closed point Chapter 3 made about GetPluginCapabilities applies here unchanged: this is one slice literal, so supporting a new capability later — if a future chapter adds one — means adding an entry to it, not restructuring how this method works or rewriting this test.

Where a node's identity actually comes from

NodeGetInfo is done, but it's currently only as correct as the string you hand NewDriver, and main.go doesn't have a real one yet. The obvious first instinct is os.Hostname() — Go's standard library already knows the machine's hostname, so why not just ask it?

Because inside a Kubernetes pod, "the machine's hostname" and "the node's hostname" are two different things. By default, a container's hostname is set to its pod's name, not the underlying node's — os.Hostname() inside your driver container would return something like localdir-csi-node-4kx9p, not the actual node kubelet is running on. Feed that into NodeGetInfo, and the node ID Kubernetes stores for this node is actually a pod name that gets deleted and replaced the next time this pod restarts — exactly the kind of subtly wrong value that works fine in a five-minute test and quietly breaks something later.

The real node name lives in the Kubernetes API — specifically, in the Pod object's own spec.nodeName field, filled in by the scheduler once it decides which node a pod runs on. Getting that value into your container is what the Downward API is for: a mechanism for exposing facts Kubernetes already knows about a pod — its name, its namespace, and yes, the node it landed on — as environment variables or files inside that pod's own containers, without your code ever having to ask the Kubernetes API directly. You declare which fact you want in the pod spec; kubelet fills in the value at container start.

Add this to the localdir-csi container in deploy/node.yaml:

        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data

fieldPath: spec.nodeName is the Downward API asking for exactly that one field off this pod's own spec; NODE_NAME is just the environment variable name we're choosing to expose it under inside the container — your driver reads it with an ordinary os.Getenv("NODE_NAME"), no different from reading any other environment variable.

Update cmd/localdir-csi/main.go:

package main

import (
	"flag"
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
)

func main() {
	endpoint := flag.String("endpoint", "unix:///csi/csi.sock",
		"gRPC endpoint the driver listens on, as a unix:// URI")
	dataDir := flag.String("data-dir", "/data",
		"directory the health check reads and writes to confirm local storage is usable")
	flag.Parse()

	socketPath, err := driver.SocketPathFromEndpoint(*endpoint)
	if err != nil {
		log.Fatalf("invalid -endpoint: %v", err)
	}

	nodeID := os.Getenv("NODE_NAME")
	if nodeID == "" {
		// NODE_NAME is set by the Downward API in deploy/node.yaml, so
		// this branch never runs in the cluster. It exists purely so
		// `make run`, on your own laptop, still has something sensible
		// to report — your laptop genuinely doesn't have a Kubernetes
		// node name, so falling back to its real hostname is the closest
		// honest answer available locally.
		hostname, err := os.Hostname()
		if err != nil {
			log.Fatalf("NODE_NAME not set, and failed to read hostname as a fallback: %v", err)
		}
		nodeID = hostname
	}

	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: *dataDir},
		nodeID,
	)

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s (node %s)", socketPath, nodeID)
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

Two changes below the flags, both small: nodeID is resolved once, right after the flags are parsed, with the laptop fallback described above; and csi.RegisterNodeServer(server, d) sits right next to csi.RegisterIdentityServer(server, d) — the same one-line-per-service pattern established in Chapter 3, extended by exactly one line now that Driver answers a second service.

Redeploying

Nothing about the DaemonSet's shape changed — no new container, no new mount — so the same commands from Chapter 4 apply:

make deploy

kubectl get pods -l app=localdir-csi -w

Give it a few seconds. Where Chapter 4 showed RESTARTS climbing on the node-driver-registrar container, it should now hold steady at 0 — kubelet's NodeGetInfo call finally has something real to answer with, registration succeeds, and there's nothing left to crash-loop over.

Confirm it the same way Chapter 4 confirmed the absence — by checking for the presence, this time:

kubectl get nodes
kubectl get csinode <node-name> -o yaml

Under spec.drivers, you should now find an entry for localdir.csi.example.com, with nodeID set to whatever your kind node is actually named. That's the same object, the same field, that Chapter 4 checked and came up empty — now populated, because the one piece of information kubelet was missing finally exists.

What you should have now

• internal/driver/driver.go with Driver embedding both csi.UnimplementedIdentityServer and csi.UnimplementedNodeServer, and a nodeID field threaded through the constructor as a plain string, not an interface
• internal/driver/node.go implementing NodeGetInfo and NodeGetCapabilities, both tested before they were written
• cmd/localdir-csi/main.go resolving a real node ID from the NODE_NAME environment variable — set via the Kubernetes Downward API in deploy/node.yaml — with a hostname-based fallback for local runs, and registering the Node service alongside Identity
• A driver pod in your kind cluster that kubelet now fully trusts, confirmed by a CSINode entry that didn't exist at the end of Chapter 4

Two methods of csi.NodeServer are implemented; the two that actually matter for a pod to use a volume — NodePublishVolume and NodeUnpublishVolume, the calls that turn a mounted directory into something a container can read and write through — are next, in Chapter 6.

Chapter 6: The Node Service, Part 2

NodeGetInfo and NodeGetCapabilities answer questions. Neither one touches a filesystem, mounts anything, or changes what a pod can actually see. This chapter writes the two methods that do: NodePublishVolume and NodeUnpublishVolume — the calls that turn a directory of data sitting on a node's disk into something a container can read and write through, and then, later, take that access away again cleanly. This is the payoff Chapter 5 pointed at, and it's also where mount --bind, first mentioned back in Chapter 2, stops being a phrase in a sidebar and becomes something your own code calls.

What "publish" actually means here

A bind mount is a Linux feature that makes one directory also appear at a second path — not a copy, the same files, reachable through two different doors. If /var/lib/localdir-csi/data/vol-1 holds a volume's real data, bind-mounting it onto some other path makes that second path behave exactly like the first: read a file through either one, and you're reading the same bytes.

That's exactly the tool NodePublishVolume needs. Kubernetes decides where a container should see its volume — a path deep inside kubelet's own directory structure, specific to one pod, that your driver doesn't get to choose — and hands you that path as target_path. Your volume's actual data already lives somewhere else, at a location only your driver knows about. A bind mount is how you make the data at your location appear, unmodified, at Kubernetes's location. "Publish" means "make this volume visible at the path you were told to use." "Unpublish" means undo exactly that, leaving the underlying data untouched.

One naming note before the code: the CSI spec itself, and its own generated Go doc comments, refer to whatever's driving these calls as the CO — short for Container Orchestrator. Kubernetes is the only CO this book ever talks to, but the spec is deliberately written so a driver like this one could just as easily plug into a different orchestrator without changing a line of RPC code, which is why "CO" shows up instead of "Kubernetes" in the spec's own text, and in this chapter's code comments.

Chapter 2 already flagged why this can't be tested by just running things on your own laptop: kind nodes are real Linux containers, so when localdir-csi calls mount --bind inside one, it's genuinely mounting something, inside that container's own filesystem — the same kind of operation, on the same kind of system, that would happen on a real cloud VM. That's also why this chapter's hands-on testing happens by reaching into a kind node directly, the way Chapter 2 said it eventually would.

A narrow interface for the one thing we need: Mounter

Performing a real bind mount from Go means calling into k8s.io/mount-utils, the same library Kubernetes's own CSI drivers use — it wraps the underlying mount/umount system calls and the bookkeeping needed to ask "is this path already a mount point?" correctly on Linux. Its Interface type exposes close to a dozen methods (variations on Mount for different flag combinations, List, GetMountRefs, and more). Driver needs exactly three of them: mount something, unmount something, and ask whether a path is currently a mount point.

Rather than depending on the whole real interface, declare a smaller one that says exactly that:

type Mounter interface {
	Mount(source, target, fsType string, options []string) error
	Unmount(target string) error
	IsMountPoint(target string) (bool, error)
}

This is the Interface Segregation Principle, the same family of idea as HealthChecker from Chapter 3: depend on the smallest interface that describes what you actually need, not the largest one a library happens to offer. The payoff here is concrete, not just tidy — because Go decides whether a type satisfies an interface by comparing method signatures, not by any explicit "implements" declaration, the real value k8s.io/mount-utils hands back already has a Mount, an Unmount, and an IsMountPoint with these exact signatures. It satisfies Mounter automatically. No adapter, no wrapper type, no glue code — declaring the interface Driver actually needs is the entire integration.

Driver gains a data directory and a mounter

NodePublishVolume needs to know two new things: where on this node volumes actually live, and something that can perform a mount. Update internal/driver/driver.go:

package driver

import (
	"github.com/container-storage-interface/spec/lib/go/csi"
)

// HealthChecker describes anything that can report whether the driver's
// backend is usable right now. Identity's Probe method depends on this
// interface — not on any specific way of checking health — so that a
// real filesystem check and a test fake are equally valid things to hand
// it. This is the whole idea behind Dependency Inversion: Driver names
// what it needs, not how that need gets satisfied.
type HealthChecker interface {
	Healthy() bool
}

// Mounter describes the three mount operations Driver actually performs.
// A real *mount-utils mounter satisfies this automatically; tests hand it
// a fake that never touches a real filesystem.
type Mounter interface {
	Mount(source, target, fsType string, options []string) error
	Unmount(target string) error
	IsMountPoint(target string) (bool, error)
}

// Driver holds everything our CSI driver needs to answer gRPC calls.
type Driver struct {
	csi.UnimplementedIdentityServer
	csi.UnimplementedNodeServer

	name    string
	version string
	health  HealthChecker
	nodeID  string
	dataDir string
	mount   Mounter
}

// NewDriver builds a Driver. health and mount can be nil — health is
// guarded in Probe, and mount is only touched by RPCs that need it —
// which is useful for tests that don't care about either, but main.go
// always passes real implementations of both.
func NewDriver(name, version string, health HealthChecker, nodeID, dataDir string, mount Mounter) *Driver {
	return &Driver{
		name:    name,
		version: version,
		health:  health,
		nodeID:  nodeID,
		dataDir: dataDir,
		mount:   mount,
	}
}

// Compile-time checks: these lines do nothing at runtime. They exist
// purely so that if you ever change Driver in a way that breaks either
// interface, the build fails immediately with a clear error — instead of
// failing later, mysteriously, when gRPC tries to register your service.
var _ csi.IdentityServer = (*Driver)(nil)
var _ csi.NodeServer = (*Driver)(nil)

NewDriver grew two more parameters, and — the same lesson every constructor change in this book has taught — that breaks every existing call site at once. Run the tests:

go test ./internal/driver/...

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
internal/driver/identity_test.go:10:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/identity_test.go:26:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/identity_test.go:66:97: not enough arguments in call to NewDriver
	have (string, string, *fakeHealthChecker, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/node_test.go:10:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/node_test.go:23:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]
FAIL

None of these five existing tests care about a data directory or a mounter — Identity's tests never touch either, and NodeGetInfo/ NodeGetCapabilities only ever read nodeID. The smallest honest fix would be a placeholder for each — an empty string for dataDir, nil for mount — the same way nil already stands in for a health checker these particular tests don't need. But this is the third time NewDriver has grown a parameter and broken every call site at once, and this chapter is about to add a dozen more tests that all need to build a Driver the same way. Patching five call sites by hand, again, is the smallest fix for today; it isn't the smallest fix for the pattern.

Create internal/driver/testing_test.go — a file with no tests of its own, only shared scaffolding every other test file in this package can call:

package driver

import (
	"testing"
)

// newTestDriver builds a Driver wired to the package's shared test
// constants, with the given dataDir and mount. Pass "" for dataDir when
// a test doesn't need one, and nil for mount when the call path never
// reaches the mounter.
func newTestDriver(t *testing.T, dataDir string, mount Mounter) *Driver {
	t.Helper()
	return NewDriver(name, version, nil, nodeID, dataDir, mount)
}

// newTestDriverInTempDir is the common case: a fresh temp dataDir and no
// mounter. Each call gets its own t.TempDir(), so tests stay isolated
// from each other.
func newTestDriverInTempDir(t *testing.T) *Driver {
	t.Helper()
	return newTestDriver(t, t.TempDir(), nil)
}

newTestDriver hardcodes health to nil — every test that actually needs to control health, like TestProbe, still calls NewDriver directly, exactly as before. What it does fix is the other five parameters: name, version, and nodeID are always the same three values across this entire package, and dataDir/mount are the two that actually vary test to test. t.Helper() matters here for the same reason it always does — a future failure inside a test built this way should point at that test's own line, not at a line inside this helper.

name, version, and nodeID need to actually exist somewhere now. Add this to internal/driver/identity_test.go, alongside its existing imports:

const (
	name    = "localdir.csi.example.com"
	version = "0.1.0"
	nodeID  = "test-node-1"
	dataDir = ""
)

These replace the literal strings scattered across every test file in this package — the same "localdir.csi.example.com", "0.1.0", and node ID that TestGetPluginInfo, TestGetPluginCapabilities, TestProbe, TestNodeGetInfo, and TestNodeGetCapabilities were each already retyping by hand. Update the five call sites to match: the four that don't need a real health checker become

d := newTestDriver(t, dataDir, nil)

and TestProbe's, which still needs to hand Driver a real *fakeHealthChecker, becomes

d := NewDriver(name, version, &fakeHealthChecker{healthy: tt.healthy}, nodeID, dataDir, nil)

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green again — and if NewDriver ever grows another parameter, only newTestDriver itself needs to change. Every test that calls it stays exactly as it is; this is newTestDriver's whole reason for existing.

Writing NodePublishVolume, test-first

Testing this honestly means never touching a real filesystem mount — that needs root, and a test suite that only passes as root isn't one you can trust in CI. Mounter exists precisely so a test can hand Driver a fake that records what it was asked to do instead of doing it. One test at a time, same rhythm as every RPC so far: write a test, watch it fail, write only enough code to pass it, then move to the next case.

Add this to internal/driver/node_test.go, alongside the existing NodeGetInfo/NodeGetCapabilities tests:

type mountCall struct {
	source, target, fsType string
	options                []string
}

type fakeMounter struct {
	mounted         bool
	isMountPointErr error
	mountErr        error
	unmountErr      error

	mountCalls   []mountCall
	unmountCalls []string
}

func (f *fakeMounter) Mount(source, target, fsType string, options []string) error {
	f.mountCalls = append(f.mountCalls, mountCall{source, target, fsType, options})
	return f.mountErr
}

func (f *fakeMounter) Unmount(target string) error {
	f.unmountCalls = append(f.unmountCalls, target)
	return f.unmountErr
}

func (f *fakeMounter) IsMountPoint(target string) (bool, error) {
	if f.isMountPointErr != nil {
		return false, f.isMountPointErr
	}
	return f.mounted, nil
}

fakeMounter belongs in node_test.go — nothing outside Node's own tests needs it. The next piece doesn't: every test below needs one particular *csi.VolumeCapability, and later chapters' Controller tests will need the exact same one. That makes it shared scaffolding, not node_test.go's private business — add it to internal/driver/testing_test.go instead:

func mountCapability() *csi.VolumeCapability {
	return &csi.VolumeCapability{
		AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
		AccessMode: &csi.VolumeCapability_AccessMode{
			Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
		},
	}
}

Add "github.com/container-storage-interface/spec/lib/go/csi" to testing_test.go's imports.

mountCapability is a test-only helper — nothing in Driver itself calls it — that builds the one piece of a valid request every test below needs and none of them should have to construct by hand: a *csi.VolumeCapability. The CSI spec uses VolumeCapability to describe how a volume needs to be usable, and it's built from two parts. AccessType is a choice between two shapes: a Block capability (the volume shows up as a raw block device, no filesystem at all) or a Mount capability (the volume shows up as a directory you mount). This driver only ever deals in directories, so the helper always builds the Mount variant, and leaves the nested MountVolume value empty — it has fields for a filesystem type and mount flags, neither of which apply here, since every mount this driver performs is a plain bind mount rather than a real filesystem format.

AccessMode describes how many things are allowed to use the volume at once, and how. SINGLE_NODE_WRITER means one node, read-write — the simplest case, and the only one NodePublishVolume assumes right now. The spec defines several other modes, including read-only and multiple-nodes-at-once variants, but nothing in this chapter's code inspects AccessMode's actual value yet: the validation check below only asks whether the VolumeCapability pointer is nil at all, since volume_capability is a required field of the request regardless of which mode it names. AccessMode starts to matter once a real StorageClass and PersistentVolumeClaim can specify one.

This is also the first RPC in the book that needs more than one invalid-request case checked the same way — and it won't be the last; NodeUnpublishVolume, a few pages from now, needs the same shape of test, and so will several Controller methods in Chapter 7. Rather than writing "build a table, loop over it, assert InvalidArgument" by hand every time, add two small pieces of shared scaffolding to internal/driver/testing_test.go:

// requireStatusCode asserts that err carries the given gRPC code. req is
// included in the failure message for debugging.
func requireStatusCode(t *testing.T, err error, want codes.Code, req any) {
	t.Helper()
	if got := status.Code(err); got != want {
		t.Fatalf("code = %v, want %v (req=%+v)", got, want, req)
	}
}

// validationCase is one row of a table-driven validation test: a request
// that should be rejected with codes.InvalidArgument.
type validationCase[T any] struct {
	name string
	req  T
}

// runValidation runs a table of validation cases against call. For each
// case it builds a fresh driver via newDriver, invokes call, and asserts
// the error is InvalidArgument. newDriver is called inside the subtest,
// so per-test temp dirs and contexts stay scoped to that subtest.
func runValidation[T any](
	t *testing.T,
	cases []validationCase[T],
	newDriver func(t *testing.T) *Driver,
	call func(t *testing.T, d *Driver, req T) error,
) {
	t.Helper()
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			d := newDriver(t)
			err := call(t, d, tc.req)
			requireStatusCode(t, err, codes.InvalidArgument, tc.req)
		})
	}
}

Add "google.golang.org/grpc/codes" and "google.golang.org/grpc/status" to testing_test.go's imports.

requireStatusCode replaces the one-off if status.Code(err) != want { t.Fatalf(...) } block every RPC's tests were about to start repeating. validationCase/runValidation go one step further: [T any] is a Go generic — this pair works for *csi.NodePublishVolumeRequest, *csi.NodeUnpublishVolumeRequest, or any other request type, without writing a new loop for each. That's the same Open/Closed shape Chapter 3 already applied to a slice of capabilities, applied here to a slice of test cases instead: a new RPC's validation test becomes a new table and a two-line call to runValidation, not a new copy of the loop.

With that in place, TestNodePublishVolume_Validation itself gets much smaller. Add this to internal/driver/node_test.go, alongside the existing NodeGetInfo/NodeGetCapabilities tests:

func TestNodePublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.NodePublishVolumeRequest]{
		{
			name: "missing volume ID",
			req: &csi.NodePublishVolumeRequest{
				TargetPath:       "/target",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing target path",
			req: &csi.NodePublishVolumeRequest{
				VolumeId:         "vol-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing volume capability",
			req: &csi.NodePublishVolumeRequest{
				VolumeId:   "vol-1",
				TargetPath: "/target",
			},
		},
	}
	runValidation(t, cases,
		func(t *testing.T) *Driver { return newTestDriver(t, dataDir, &fakeMounter{}) },
		func(t *testing.T, d *Driver, req *csi.NodePublishVolumeRequest) error {
			_, err := d.NodePublishVolume(t.Context(), req)
			return err
		},
	)
}

This needs google.golang.org/grpc/codes in node_test.go's imports too — runValidation takes care of status, but node_test.go still names codes.NotFound and friends directly in tests further down (path/filepath and os are coming, but not yet).

Run it:

go test ./internal/driver/...

Red — and a different shape of red than any RPC so far. NodePublishVolume doesn't exist as a method on *Driver yet, so these calls fall through to the embedded UnimplementedNodeServer's stub, which returns a real gRPC status error carrying codes.Unimplemented — not the code this test wants, InvalidArgument, but a real, specific one all the same, not a generic failure. Notice the failure points at testing_test.go, not node_test.go — requireStatusCode calls t.Helper(), but the subtest closure inside runValidation that calls it doesn't, so Go attributes the failure to the innermost non-helper frame, which lives in testing_test.go:

--- FAIL: TestNodePublishVolume_Validation (0.00s)
    --- FAIL: TestNodePublishVolume_Validation/missing_volume_ID (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: PublishContext:map[] StagingTargetPath: TargetPath:/target VolumeCapability:0xc00010c1e0 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestNodePublishVolume_Validation/missing_target_path (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath: VolumeCapability:0xc00010c1f8 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestNodePublishVolume_Validation/missing_volume_capability (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath:/target VolumeCapability:<nil> Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

(The exact line number inside testing_test.go, and the exact pointer address printed for VolumeCapability, will differ slightly depending on how you've laid the file out — neither one is the point; the code and the message are.)

Write just enough of NodePublishVolume to fix it. status.Error is new this chapter — every RPC before now either succeeded or fell through to UnimplementedNodeServer's stub, never returning a specific failure code on purpose. The CSI spec ties specific conditions to specific gRPC status codes (InvalidArgument for a malformed request), and status.Error(code, message) is how a handler attaches a real one instead of a generic error. Create internal/driver/node.go's new method:

func (d *Driver) NodePublishVolume(
	ctx context.Context,
	req *csi.NodePublishVolumeRequest,
) (*csi.NodePublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetTargetPath() == "" {
		return nil, status.Error(codes.InvalidArgument, "target_path is required")
	}
	if req.GetVolumeCapability() == nil {
		return nil, status.Error(codes.InvalidArgument, "volume_capability is required")
	}

	return &csi.NodePublishVolumeResponse{}, nil
}

That last line is deliberately a stub — the three checks are all this step needs, and returning success unconditionally afterward is the smallest thing that makes the current test pass. Add "google.golang.org/grpc/codes" and "google.golang.org/grpc/status" to node.go's imports too.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green. Next case: a volume that doesn't exist. Add this test below TestNodePublishVolume_Validation:

func TestNodePublishVolume_VolumeNotFound(t *testing.T) {
	d := newTestDriver(t, t.TempDir(), &fakeMounter{})

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "does-not-exist",
		TargetPath:       filepath.Join(t.TempDir(), "target"),
		VolumeCapability: mountCapability(),
	}

	_, err := d.NodePublishVolume(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

Add "path/filepath" to node_test.go's imports.

go test ./internal/driver/...

Red — the stub from the last step returns success unconditionally, so there's no NotFound to find yet. This time requireStatusCode is called directly from TestNodePublishVolume_VolumeNotFound itself, no subtest closure in between, so the failure points right back at node_test.go, at requireStatusCode's own call site:

--- FAIL: TestNodePublishVolume_VolumeNotFound (0.00s)
    node_test.go:113: code = OK, want NotFound (req=&{VolumeId:does-not-exist PublishContext:map[] StagingTargetPath: TargetPath:/tmp/TestNodePublishVolume_VolumeNotFound2786629652/002/target VolumeCapability:0xc00010c2e8 Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.005s
FAIL

(As ever, the exact temp path and pointer address in your own output will differ from this one.)

Fix it. This is where the source path convention this chapter already described gets written down in code: filepath.Join(d.dataDir, req.GetVolumeId()) assumes a volume's data lives in a directory named after its own ID, directly under the driver's data root — right now that directory has to exist already, or the request fails with NotFound; Chapter 7's CreateVolume is what will make it exist automatically, using this exact same naming rule.

	source := filepath.Join(d.dataDir, req.GetVolumeId())
	if _, err := os.Stat(source); err != nil {
		if os.IsNotExist(err) {
			return nil, status.Errorf(codes.NotFound, "volume %q not found", req.GetVolumeId())
		}
		return nil, status.Errorf(codes.Internal, "checking volume %q: %v", req.GetVolumeId(), err)
	}

	return &csi.NodePublishVolumeResponse{}, nil

That replaces the old stub return — insert it right after the three validation checks in NodePublishVolume, and add "os" and "path/filepath" to node.go's imports.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green. Next: actually mounting something, for a volume that does exist — TestNodePublishVolume_MountsTheVolume, added below TestNodePublishVolume_VolumeNotFound. Setting up "a fake volume directory" is also something every remaining test in this file, and every Controller test Chapter 7 adds, needs to do at least once. Add one more helper to internal/driver/testing_test.go first:

// makeVolumeDir creates a fake volume directory named volName inside
// dataDir and returns its path. Fails the test immediately if the mkdir
// fails.
func makeVolumeDir(t *testing.T, dataDir, volName string) string {
	t.Helper()
	p := filepath.Join(dataDir, volName)
	if err := os.Mkdir(p, 0750); err != nil {
		t.Fatalf("creating fake volume dir %q: %v", p, err)
	}
	return p
}

Add "os" and "path/filepath" to testing_test.go's imports.

func TestNodePublishVolume_MountsTheVolume(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	mounter := &fakeMounter{}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	if _, err := os.Stat(targetPath); err != nil {
		t.Fatalf("target path was not created: %v", err)
	}

	if len(mounter.mountCalls) != 1 {
		t.Fatalf("got %d Mount calls, want 1", len(mounter.mountCalls))
	}
	got := mounter.mountCalls[0]
	wantSource := filepath.Join(dataDir, "vol-1")
	if got.source != wantSource {
		t.Errorf("source = %q, want %q", got.source, wantSource)
	}
	if got.target != targetPath {
		t.Errorf("target = %q, want %q", got.target, targetPath)
	}
	if len(got.options) != 1 || got.options[0] != "bind" {
		t.Errorf("options = %v, want [bind]", got.options)
	}
}

Add "os" to node_test.go's imports too.

go test ./internal/driver/...

Red — NodePublishVolume finds the volume now, but still just returns success without creating the target path or calling Mount at all (t.TempDir() generates a fresh, uniquely-named directory on every run, so the exact path in your own output will differ from this one):

--- FAIL: TestNodePublishVolume_MountsTheVolume (0.00s)
    node_test.go:135: target path was not created: stat /tmp/TestNodePublishVolume_MountsTheVolume3795384414/002/target: no such file or directory
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it — the spec makes creating target_path the driver's job, not the CO's, so NodePublishVolume does that unconditionally before touching the mounter:

	// The spec makes creating target_path our job, not the CO's.
	if err := os.MkdirAll(req.GetTargetPath(), 0750); err != nil {
		return nil, status.Errorf(codes.Internal, "creating target path: %v", err)
	}

	if err := d.mount.Mount(source, req.GetTargetPath(), "", []string{"bind"}); err != nil {
		return nil, status.Errorf(codes.Internal, "mounting volume %q: %v", req.GetVolumeId(), err)
	}

	return &csi.NodePublishVolumeResponse{}, nil

Replace the old return &csi.NodePublishVolumeResponse{}, nil that followed the NotFound check with this.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green. Next: honoring readonly. Add this below TestNodePublishVolume_MountsTheVolume:

func TestNodePublishVolume_Readonly(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	mounter := &fakeMounter{}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         true,
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	got := mounter.mountCalls[0].options
	if len(got) != 2 || got[0] != "bind" || got[1] != "ro" {
		t.Errorf("options = %v, want [bind ro]", got)
	}
}

go test ./internal/driver/...

Red — the mount options are hardcoded to [bind] and never look at req.GetReadonly():

--- FAIL: TestNodePublishVolume_Readonly (0.00s)
    node_test.go:175: options = [bind], want [bind ro]
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it — build the options slice instead of hardcoding it:

	options := []string{"bind"}
	if req.GetReadonly() {
		options = append(options, "ro")
	}
	if err := d.mount.Mount(source, req.GetTargetPath(), "", options); err != nil {
		return nil, status.Errorf(codes.Internal, "mounting volume %q: %v", req.GetVolumeId(), err)
	}

This replaces the previous Mount call, which passed []string{"bind"} directly.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green. One case left for this method: a target that's already published. Add this below TestNodePublishVolume_Readonly:

func TestNodePublishVolume_AlreadyMounted(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := t.TempDir()

	mounter := &fakeMounter{mounted: true}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	if len(mounter.mountCalls) != 0 {
		t.Errorf("got %d Mount calls, want 0 — already-mounted target must not be remounted", len(mounter.mountCalls))
	}
}

go test ./internal/driver/...

Red — nothing so far ever asks the mounter whether the target is already mounted, so Mount gets called again even though fakeMounter.mounted is true:

--- FAIL: TestNodePublishVolume_AlreadyMounted (0.00s)
    node_test.go:198: got 1 Mount calls, want 0 — already-mounted target must not be remounted
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it by checking first, and short-circuiting if the answer is yes:

	mounted, err := d.mount.IsMountPoint(req.GetTargetPath())
	if err != nil {
		return nil, status.Errorf(codes.Internal, "checking target path: %v", err)
	}
	if mounted {
		// NodePublishVolume must be idempotent: a CO that isn't sure a
		// previous call succeeded is expected to retry it, and retrying
		// must not fail or remount.
		return &csi.NodePublishVolumeResponse{}, nil
	}

Insert this right after the os.MkdirAll call and before building the options slice.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green — and the common case is handled: the exact same request, retried. The comment above is worth pausing on: idempotent means calling an operation more than once, with the same arguments, has the same effect as calling it exactly once — nothing extra happens, and nothing errors out, the second time. That's not incidental here; the CSI spec requires it, because a CO that times out waiting for a response has no way to know whether the call actually succeeded, and the only safe thing for it to do is call again.

That phrase "with the same arguments" is doing more work than it looks like it is.

Writing NodePublishVolume, test-first: telling a retry from a conflict

IsMountPoint(target_path) only ever answers true or false. It says nothing about what is mounted there, or how. Call NodePublishVolume once for a volume with readonly: false, then again for the same volume and target path with readonly: true, and the check just written waves the second call through anyway — "already mounted" reads as "safe to report success," full stop, regardless of whether this is genuinely the same request retried or a different request that happens to reuse the same volume and target path. Not a hypothetical, either: two Pods mounting the same PVC with different access modes would trigger exactly this.

Add the test, right after TestNodePublishVolume_AlreadyMounted in internal/driver/node_test.go:

func TestNodePublishVolume_RejectsAccessModeMismatch(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	first := newTestDriver(t, dataDir, &fakeMounter{})
	firstReq := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         false,
	}
	if _, err := first.NodePublishVolume(t.Context(), firstReq); err != nil {
		t.Fatalf("first NodePublishVolume() returned an error: %v", err)
	}

	// A second driver instance, same dataDir, mounter pre-set to report
	// the target as already mounted — the same signal a CO's retry would
	// see from a real kubelet, this time asking for it read-only.
	second := newTestDriver(t, dataDir, &fakeMounter{mounted: true})
	secondReq := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         true,
	}
	_, err := second.NodePublishVolume(t.Context(), secondReq)
	requireStatusCode(t, err, codes.AlreadyExists, secondReq)
}

go test ./internal/driver/...

Red — nothing anywhere records what a mount was actually made with, so there's nothing to compare the second call against:

=== RUN   TestNodePublishVolume_RejectsAccessModeMismatch
    node_test.go:229: code = OK, want AlreadyExists (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath:/tmp/TestNodePublishVolume_RejectsAccessModeMismatch4183913702/002/target VolumeCapability:0xc0000d0090 Readonly:true Secrets:map[] VolumeContext:map[]})
--- FAIL: TestNodePublishVolume_RejectsAccessModeMismatch (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.002s
FAIL

Fixing it means answering a question the driver currently has no way to ask: what was this volume actually published with, last time? A boolean "is something mounted here" isn't enough — the driver needs to remember the actual parameters. That means writing something down: a small metadata file, sitting next to the volume's data but never inside it, recording what a call actually did so a later call can compare against that record instead of guessing from a side effect. Add this to internal/driver/node.go:

// nodePublishMetaPath is where NodePublishVolume records which target path
// and access mode a volume was last published with on this node, so a
// later NodePublishVolume call for the same volume can tell a genuine
// idempotent retry (same target path, same readonly flag) apart from a
// real conflict (a different one).
func nodePublishMetaPath(dataDir, volumeID string) string {
	return filepath.Join(dataDir, ".csi-meta", "publish", volumeID+".json")
}

type nodePublishMeta struct {
	TargetPath string `json:"target_path"`
	Readonly   bool   `json:"readonly"`
}

func writeNodePublishMeta(dataDir, volumeID string, meta nodePublishMeta) error {
	metaBytes, err := json.Marshal(meta)
	if err != nil {
		return err
	}
	metaPath := nodePublishMetaPath(dataDir, volumeID)
	if err := os.MkdirAll(filepath.Dir(metaPath), 0o750); err != nil {
		return err
	}
	return os.WriteFile(metaPath, metaBytes, 0o640)
}

func readNodePublishMeta(dataDir, volumeID string) (nodePublishMeta, error) {
	var meta nodePublishMeta
	data, err := os.ReadFile(nodePublishMetaPath(dataDir, volumeID))
	if err != nil {
		return meta, err
	}
	err = json.Unmarshal(data, &meta)
	return meta, err
}

Add "encoding/json" to node.go's imports. Then change NodePublishVolume's already-mounted branch, in the same file:

	if mounted {
		// NodePublishVolume must be idempotent: a CO that isn't sure a
		// previous call succeeded is expected to retry it, and retrying
		// must not fail or remount. But "already mounted" only means
		// "safe to report success" if it was mounted with these same
		// parameters — otherwise this is a real conflict, not a retry.
		if existing, err := readNodePublishMeta(d.dataDir, req.GetVolumeId()); err == nil {
			if existing.TargetPath != req.GetTargetPath() || existing.Readonly != req.GetReadonly() {
				return nil, status.Errorf(codes.AlreadyExists, "volume %q is already published with different parameters", req.GetVolumeId())
			}
		} else if !os.IsNotExist(err) {
			return nil, status.Errorf(codes.Internal, "reading volume %q publish metadata: %v", req.GetVolumeId(), err)
		}
		return &csi.NodePublishVolumeResponse{}, nil
	}

And record the metadata right after a real mount succeeds, before NodePublishVolume's final return:

	if err := writeNodePublishMeta(d.dataDir, req.GetVolumeId(), nodePublishMeta{
		TargetPath: req.GetTargetPath(),
		Readonly:   req.GetReadonly(),
	}); err != nil {
		return nil, status.Errorf(codes.Internal, "recording volume %q publish metadata: %v", req.GetVolumeId(), err)
	}

NodeUnpublishVolume, written a little further down this chapter, gets one more line of its own to clean this metadata file back up — covered when that method's fix comes up.

go test ./internal/driver/...

=== RUN   TestNodePublishVolume_RejectsAccessModeMismatch
--- PASS: TestNodePublishVolume_RejectsAccessModeMismatch (0.00s)

Green — and now NodePublishVolume is done, in the stronger sense the spec actually asks for. Worth naming the pattern explicitly, because it's going to come up again: an operation that's supposed to be idempotent is not automatically safe just because the primitive underneath it happens not to fail on a repeat call — IsMountPoint returning true here, but the same shape shows up anywhere a driver checks "does this already exist" instead of "does this already exist as this specific thing." "Looks idempotent" and "is idempotent" are different claims. The first is free. The second costs one metadata file and one comparison, every time: record what you actually did, compare what's being asked against what you recorded, and only then decide whether this is the same request again or a different one wearing the same name. Keep an eye out for this shape — CreateVolume in Chapter 7 is about to skip it, on purpose, and pay for that choice in Chapter 9.

Writing NodeUnpublishVolume, test-first

Same rhythm, new method. Start with validation, the same shape as NodePublishVolume's first test. Add this below TestNodePublishVolume_AlreadyMounted:

func TestNodeUnpublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.NodeUnpublishVolumeRequest]{
		{name: "missing volume ID", req: &csi.NodeUnpublishVolumeRequest{TargetPath: "/target"}},
		{name: "missing target path", req: &csi.NodeUnpublishVolumeRequest{VolumeId: "vol-1"}},
	}
	runValidation(t, cases,
		func(t *testing.T) *Driver { return newTestDriver(t, dataDir, &fakeMounter{}) },
		func(t *testing.T, d *Driver, req *csi.NodeUnpublishVolumeRequest) error {
			_, err := d.NodeUnpublishVolume(t.Context(), req)
			return err
		},
	)
}

go test ./internal/driver/...

Red — NodeUnpublishVolume is still only the promoted UnimplementedNodeServer stub, the same Unimplemented-code shape NodePublishVolume's first test hit. And the same testing_test.go attribution as before, for the same reason:

--- FAIL: TestNodeUnpublishVolume_Validation (0.00s)
    --- FAIL: TestNodeUnpublishVolume_Validation/missing_volume_ID (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: TargetPath:/target})
    --- FAIL: TestNodeUnpublishVolume_Validation/missing_target_path (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 TargetPath:})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Write just enough to fix it — the same two-check, stub-return shape NodePublishVolume started with:

func (d *Driver) NodeUnpublishVolume(
	ctx context.Context,
	req *csi.NodeUnpublishVolumeRequest,
) (*csi.NodeUnpublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetTargetPath() == "" {
		return nil, status.Error(codes.InvalidArgument, "target_path is required")
	}

	return &csi.NodeUnpublishVolumeResponse{}, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green. Next: actually unmounting and cleaning up. Add this below TestNodeUnpublishVolume_Validation:

func TestNodeUnpublishVolume_UnmountsAndRemoves(t *testing.T) {
	targetPath := filepath.Join(t.TempDir(), "target")
	if err := os.Mkdir(targetPath, 0750); err != nil {
		t.Fatalf("setting up fake target path: %v", err)
	}

	mounter := &fakeMounter{mounted: true}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodeUnpublishVolumeRequest{
		VolumeId:   "vol-1",
		TargetPath: targetPath,
	}

	if _, err := d.NodeUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodeUnpublishVolume() returned an error: %v", err)
	}

	if len(mounter.unmountCalls) != 1 || mounter.unmountCalls[0] != targetPath {
		t.Errorf("unmountCalls = %v, want [%q]", mounter.unmountCalls, targetPath)
	}

	if _, err := os.Stat(targetPath); !os.IsNotExist(err) {
		t.Errorf("target path still exists after NodeUnpublishVolume")
	}
}

go test ./internal/driver/...

Red — the stub from the last step returns success without ever calling Unmount or removing anything (again, the exact temp path in your output will be different from this one):

--- FAIL: TestNodeUnpublishVolume_UnmountsAndRemoves (0.00s)
    node_test.go:235: unmountCalls = [], want ["/tmp/TestNodeUnpublishVolume_UnmountsAndRemoves1008210749/001/target"]
    node_test.go:239: target path still exists after NodeUnpublishVolume
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

Fix it. The CSI spec's own text for this method says the driver "MUST delete the file or directory it created" at target_path — the os.RemoveAll below isn't tidiness, it's a spec requirement:

	if err := d.mount.Unmount(req.GetTargetPath()); err != nil {
		return nil, status.Errorf(codes.Internal, "unmounting %q: %v", req.GetTargetPath(), err)
	}

	if err := os.RemoveAll(req.GetTargetPath()); err != nil {
		return nil, status.Errorf(codes.Internal, "removing target path: %v", err)
	}

	// Clean up the publish metadata NodePublishVolume recorded, so a
	// volume republished later starts with no stale record of what an
	// earlier, now-unpublished call did.
	if err := os.Remove(nodePublishMetaPath(d.dataDir, req.GetVolumeId())); err != nil && !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "deleting volume %q publish metadata: %v", req.GetVolumeId(), err)
	}

	return &csi.NodeUnpublishVolumeResponse{}, nil

This replaces the old return &csi.NodeUnpublishVolumeResponse{}, nil stub, right after the two validation checks — and closes the loop on the metadata file NodePublishVolume started writing earlier in this chapter.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.007s

Green. Last case: idempotency, the same property NodePublishVolume needed. Add this below TestNodeUnpublishVolume_UnmountsAndRemoves:

func TestNodeUnpublishVolume_NotMounted(t *testing.T) {
	mounter := &fakeMounter{isMountPointErr: os.ErrNotExist}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodeUnpublishVolumeRequest{
		VolumeId:   "vol-1",
		TargetPath: filepath.Join(t.TempDir(), "already-gone"),
	}

	if _, err := d.NodeUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodeUnpublishVolume() returned an error: %v", err)
	}

	if len(mounter.unmountCalls) != 0 {
		t.Errorf("got %d Unmount calls, want 0 — nothing was mounted", len(mounter.unmountCalls))
	}
}

go test ./internal/driver/...

Red — NodeUnpublishVolume still calls Unmount unconditionally, so it does even when the fake reports the target was never there in the first place:

--- FAIL: TestNodeUnpublishVolume_NotMounted (0.00s)
    node_test.go:257: got 1 Unmount calls, want 0 — nothing was mounted
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it the same way NodePublishVolume learned to ask first:

	mounted, err := d.mount.IsMountPoint(req.GetTargetPath())
	if err != nil {
		if os.IsNotExist(err) {
			// Nothing left to clean up — this is what a retried call
			// after a previous, fully-successful one looks like.
			return &csi.NodeUnpublishVolumeResponse{}, nil
		}
		return nil, status.Errorf(codes.Internal, "checking target path: %v", err)
	}

	if mounted {
		if err := d.mount.Unmount(req.GetTargetPath()); err != nil {
			return nil, status.Errorf(codes.Internal, "unmounting %q: %v", req.GetTargetPath(), err)
		}
	}

Replace the unconditional Unmount call with this — the os.RemoveAll call that follows stays exactly as it was.

go test ./internal/driver/... -v

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
--- PASS: TestNodeGetInfo (0.00s)
=== RUN   TestNodeGetCapabilities
--- PASS: TestNodeGetCapabilities (0.00s)
=== RUN   TestNodePublishVolume_Validation
=== RUN   TestNodePublishVolume_Validation/missing_volume_ID
=== RUN   TestNodePublishVolume_Validation/missing_target_path
=== RUN   TestNodePublishVolume_Validation/missing_volume_capability
--- PASS: TestNodePublishVolume_Validation (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_volume_ID (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_target_path (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_volume_capability (0.00s)
=== RUN   TestNodePublishVolume_VolumeNotFound
--- PASS: TestNodePublishVolume_VolumeNotFound (0.00s)
=== RUN   TestNodePublishVolume_MountsTheVolume
--- PASS: TestNodePublishVolume_MountsTheVolume (0.00s)
=== RUN   TestNodePublishVolume_Readonly
--- PASS: TestNodePublishVolume_Readonly (0.00s)
=== RUN   TestNodePublishVolume_AlreadyMounted
--- PASS: TestNodePublishVolume_AlreadyMounted (0.00s)
=== RUN   TestNodeUnpublishVolume_Validation
=== RUN   TestNodeUnpublishVolume_Validation/missing_volume_ID
=== RUN   TestNodeUnpublishVolume_Validation/missing_target_path
--- PASS: TestNodeUnpublishVolume_Validation (0.00s)
    --- PASS: TestNodeUnpublishVolume_Validation/missing_volume_ID (0.00s)
    --- PASS: TestNodeUnpublishVolume_Validation/missing_target_path (0.00s)
=== RUN   TestNodeUnpublishVolume_UnmountsAndRemoves
--- PASS: TestNodeUnpublishVolume_UnmountsAndRemoves (0.00s)
=== RUN   TestNodeUnpublishVolume_NotMounted
--- PASS: TestNodeUnpublishVolume_NotMounted (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.007s

Green, and every test written across both methods still passes — eight small red/green cycles, none of them touching a real filesystem mount, adding up to the two methods that let a pod actually use a volume.

Letting the container actually mount things

A working Mounter is only half the story — the container it runs in needs the mount command itself to actually exist, needs to be allowed to call it, and needs to be able to reach the exact host path kubelet expects a volume to land at. All three are easy to get wrong silently.

k8s.io/mount-utils's Mount doesn't perform a bind mount through a Linux system call directly — it shells out to a real mount executable, the same one you'd run by typing mount at a terminal, resolved from whatever $PATH the process sees. The distroless image Chapter 4 built the final container stage from was chosen because it has nothing beyond the compiled binary and the C library it links against — no shell, no coreutils, no mount. That never mattered before this chapter, since nothing earlier ever shelled out to anything, but NodePublishVolume fails the moment it's asked to mount something for real:

mounting volume "test-vol-1": mount failed: exec: "mount": executable file not found in $PATH

Fix it by building the final image from something that actually ships a mount binary. Alpine's package manager makes this a small, deliberate addition rather than an all-or-nothing tradeoff — util-linux is the package that provides a real mount:

# Stage 2: the image that actually ships. Alpine is still small — a few
# megabytes, not a full distro — but unlike the fully binary-less
# distroless image, it can host the one external command NodePublishVolume
# genuinely needs: a real mount.
FROM alpine:3.24
RUN apk add --no-cache util-linux
COPY --from=build /out/localdir-csi /localdir-csi
ENTRYPOINT ["/localdir-csi"]

This replaces the Dockerfile's second stage from Chapter 4 — everything from FROM gcr.io/distroless/static-debian12 onward — while the first stage, the one that actually compiles the binary, is unchanged. That first stage's CGO_ENABLED=0 flag is exactly why the swap is safe: it produces a binary with no dynamic C library dependency at all, so it runs identically whichever C library the final image happens to carry — glibc, in distroless, or musl, in Alpine. csi-driver-host-path, the reference CSI implementation this chapter has already leaned on for real-world detail, makes the identical choice for the identical reason: its own node plugin builds on Alpine with util-linux installed, because a driver that calls mount needs mount to exist somewhere inside its own container.

Linux treats mounting — even a bind mount between two directories the calling process already owns — as a privileged operation, gated behind a capability ordinary containers don't have by default. Without it, d.mount.Mount(...) returns a permission error the instant it runs for real, regardless of how correct the surrounding Go code is. Grant it by adding a securityContext to the localdir-csi container:

        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

Separately, target_path lives under /var/lib/kubelet/pods/ on the node — kubelet's own directory, not one your driver controls, and not one it currently has any access to at all. It needs a hostPath mount for that directory too, the same pattern Chapter 4 used for the socket directory. But a plain hostPath mount isn't quite enough here, and it's worth understanding why.

Every container gets its own private view of what's currently mounted where — a mount namespace. By default, a mount --bind your driver performs inside its own container stays inside that container's mount namespace; kubelet, watching the real host filesystem from outside any container, would never see it happen. mountPropagation: Bidirectional on a volume mount widens that in both directions: mounts made inside the container become visible on the host (and to other containers sharing the mount), and mounts made on the host become visible inside the container. Without it, NodePublishVolume would report success while kubelet's own view of the filesystem never changed — a working RPC call producing an invisible result.

Add both the volume mount and the corresponding volume:

          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional

        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory

type: Directory, not DirectoryOrCreate — like registration-dir in Chapter 4, this is a path kubelet itself manages and already creates on every node it runs on; if it's missing, something more fundamental than this manifest is wrong.

The full, current deploy/node.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: localdir-csi-node
  labels:
    app: localdir-csi
spec:
  selector:
    matchLabels:
      app: localdir-csi
  template:
    metadata:
      labels:
        app: localdir-csi
    spec:
      containers:
        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/localdir.csi.example.com/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: data-dir
          hostPath:
            path: /var/lib/localdir-csi/data
            type: DirectoryOrCreate
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory

Wiring up a real Mounter in main.go

internal/driver's tests never need a real mounter, but main.go does. Pull in the library:

go get k8s.io/mount-utils

k8s.io/mount-utils declares in its own go.mod that it needs Go 1.26 or newer, and Go's module system won't let a module depend on something that requires a newer Go version than it declares itself — so this command also raises go.mod's own go line to 1.26.0. On your own machine that change is invisible: Go's toolchain manager defaults to quietly downloading whatever newer version a go.mod asks for the moment it's actually needed, so go build and go test keep working without you noticing anything happened. Inside Docker it isn't invisible, because the official golang images deliberately turn that automatic downloading off (GOTOOLCHAIN=local), so a build stays reproducible without needing network access partway through. That means the build stage of the Dockerfile from Chapter 4 needs its baked-in Go version raised to match. Update its first line:

FROM golang:1.26 AS build

Everything else in the Dockerfile is unchanged.

Update cmd/localdir-csi/main.go:

package main

import (
	"flag"
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
	mount "k8s.io/mount-utils"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
)

func main() {
	endpoint := flag.String("endpoint", "unix:///csi/csi.sock",
		"gRPC endpoint the driver listens on, as a unix:// URI")
	dataDir := flag.String("data-dir", "/data",
		"directory volumes are stored under, and the health check reads and writes to confirm local storage is usable")
	flag.Parse()

	socketPath, err := driver.SocketPathFromEndpoint(*endpoint)
	if err != nil {
		log.Fatalf("invalid -endpoint: %v", err)
	}

	nodeID := os.Getenv("NODE_NAME")
	if nodeID == "" {
		// NODE_NAME is set by the Downward API in deploy/node.yaml, so
		// this branch never runs in the cluster. It exists purely so
		// `make run`, on your own laptop, still has something sensible
		// to report — your laptop genuinely doesn't have a Kubernetes
		// node name, so falling back to its real hostname is the closest
		// honest answer available locally.
		hostname, err := os.Hostname()
		if err != nil {
			log.Fatalf("NODE_NAME not set, and failed to read hostname as a fallback: %v", err)
		}
		nodeID = hostname
	}

	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: *dataDir},
		nodeID,
		*dataDir,
		mount.New(""),
	)

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s (node %s)", socketPath, nodeID)
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

mount.New("") is k8s.io/mount-utils's own constructor for a real, Linux-backed mounter — the empty string is the path to the mount binary it shells out to for some operations, and an empty string tells it to use the default, /bin/mount. The package name is mount, which is why the import gets an explicit mount prefix here even though it's already the default — k8s.io/mount-utils is a fairly generic-sounding import path to leave unlabeled in a file that also has a field named mount on Driver itself.

*dataDir now appears twice in the NewDriver call — once building the health checker, once as Driver's own data directory. That's not duplication to clean up: LocalDirHealthChecker uses it to confirm the directory is writable at all, while Driver uses it to compute per-volume paths. Two different components, two different reasons to know the same root path.

Deploying

make deploy

kubectl get pods -l app=localdir-csi -w

Same 2/2 Running result as Chapter 5 — privileged: true and the new mountpoint-dir mount don't change anything kubelet checks during registration, so nothing about the pod's health should look different yet. What's different is what the container is now allowed to do, which the next section proves.

Proving it, inside the node

Nothing in the cluster calls NodePublishVolume automatically yet — that starts once a real pod requests a volume through a PersistentVolumeClaim, which is Chapter 8's territory. For now, calling it directly is the only way to watch it work, and Chapter 2 already told you where: inside the kind node itself, a real Linux container, the same place these bind mounts actually happen.

grpcurl isn't installed inside the node, so the first step is getting a copy of it there. Rather than guessing at which of several release archives matches the node's own CPU architecture, pull it straight out of grpcurl's own official image — a multi-architecture build that already matches whatever platform your kind node happens to be running on:

docker create --name grpcurl-tmp fullstorydev/grpcurl:v1.9.3
docker cp grpcurl-tmp:/bin/grpcurl ./grpcurl-linux
docker rm grpcurl-tmp
docker cp ./grpcurl-linux csi-dev-control-plane:/usr/local/bin/grpcurl
docker exec csi-dev-control-plane chmod +x /usr/local/bin/grpcurl

csi-dev-control-plane is the actual Docker container backing your cluster's one node — kind names it <cluster-name>-control-plane, and Chapter 2 named the cluster csi-dev.

Chapter 7 will teach CreateVolume to make a volume's data directory automatically. Until then, stand in for it by hand — this is exactly the directory structure CreateVolume will produce later, just created directly instead of through a gRPC call:

docker exec csi-dev-control-plane mkdir -p /var/lib/localdir-csi/data/test-vol-1
docker exec csi-dev-control-plane sh -c 'echo "hello from localdir-csi" > /var/lib/localdir-csi/data/test-vol-1/hello.txt'

Now call NodePublishVolume for real, against the driver's actual running socket, at a target_path shaped like the one kubelet would really choose for a pod. Every grpcurl call so far in this book has called a method that takes no arguments; this one doesn't, so it needs -d, followed by the request as JSON, using the same field names as the .proto file:

docker exec csi-dev-control-plane grpcurl -plaintext -d '{
  "volume_id": "test-vol-1",
  "target_path": "/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock csi.v1.Node/NodePublishVolume

{}

An empty response — exactly what the spec says NodePublishVolumeResponse should be, and exactly the same "success looks like nothing" shape NodeGetCapabilities's empty capability list had in Chapter 5. The proof isn't in that response; it's in whether the bind mount actually happened:

docker exec csi-dev-control-plane cat /var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount/hello.txt

hello from localdir-csi

The file you wrote at /var/lib/localdir-csi/data/test-vol-1/hello.txt is readable through an entirely different path — the one your driver was told to publish at, and the one a real container's own filesystem would be pointed at inside a pod. That's the whole chapter, proven with one cat.

Reverse it:

docker exec csi-dev-control-plane grpcurl -plaintext -d '{
  "volume_id": "test-vol-1",
  "target_path": "/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount"
}' unix:/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock csi.v1.Node/NodeUnpublishVolume

{}

docker exec csi-dev-control-plane ls /var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/

ls: cannot access '/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/': No such file or directory

Gone — NodeUnpublishVolume unmounted it and removed the directory it had created, leaving the original data at /var/lib/localdir-csi/data/test-vol-1/hello.txt completely untouched. Confirm that part too, if you want the full picture:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/test-vol-1/hello.txt

hello from localdir-csi

What you should have now

• internal/driver/driver.go: Driver carrying a dataDir and a narrow, self-declared Mounter interface — satisfied automatically by k8s.io/mount-utils's real mounter, no adapter required
• internal/driver/node.go: NodePublishVolume and NodeUnpublishVolume, both idempotent, both tested against a fake Mounter before either touched real code
• A sidecar-metadata-file pattern (nodePublishMeta, writeNodePublishMeta, readNodePublishMeta) — this book's first, recording what NodePublishVolume actually did so a later call can tell a genuine retry from a real conflict, instead of trusting IsMountPoint's bare true/false alone
• cmd/localdir-csi/main.go building a real mount.New("") and passing it, along with the data directory, into Driver
• deploy/node.yaml: the localdir-csi container running privileged, with a mountpoint-dir volume mounted Bidirectional so mounts made inside the container become visible to kubelet
• Direct, hands-on proof — a file written on one path, read back through a completely different one, inside a real kind node — that a bind mount your own code triggered actually happened

Four of csi.NodeServer's ten methods are done. What's still missing is everything upstream of them: nothing yet creates a volume's data directory, or decides how big it should be, or lets it be deleted. That's the Controller service, and it's next, in Chapter 7.

Chapter 7: The Controller Service

Every RPC so far has assumed a volume's directory already exists. NodePublishVolume looks one up at filepath.Join(d.dataDir, req.GetVolumeId()) and fails with NotFound if it isn't there — nothing before now has ever created it. That's the Controller service's job: CreateVolume and DeleteVolume, the two RPCs that make a volume exist and stop existing, answered by a completely different piece of Kubernetes than anything you've deployed yet.

Who calls Controller, and why it's a Deployment, not a DaemonSet

The Node service runs once per node because mounting is inherently a per-node fact — Chapter 1 covered that, and it's why deploy/node.yaml is a DaemonSet. Creating and deleting a volume isn't a per-node fact at all. Nothing about deciding "this volume should exist, with this much capacity" needs to happen on any particular machine, and running it on every node would just mean every node's copy racing to create the same directory. This driver's Controller service runs as an ordinary Deployment, a single replica, same as internal/driver/controller.go would run wherever that Deployment happened to land.

Nothing calls CreateVolume directly, either — not kubectl, not a person. It's called by a sidecar, external-provisioner, that Kubernetes's own storage machinery ships as a separate container image, not as anything you write. external-provisioner watches PersistentVolumeClaim objects cluster-wide, and when it sees one that asks for this driver by name, it translates that claim into a CreateVolume gRPC call against whatever's listening on the socket it shares with your Controller container — the emptyDir-socket pattern Chapter 1 described in general, and Chapter 4 named directly as exactly how this chapter would work, when it explained why node-driver-registrar needed a hostPath instead.

A concrete way to picture the whole loop: a PersistentVolumeClaim is a request card — "I need 1Gi of storage from localdir-csi" — dropped into a box that every storage driver's own clerk is watching. external-provisioner is localdir-csi's clerk, and only that driver's clerk; a claim asking for a different driver by name never reaches it at all. When it spots a card addressed to your driver, it doesn't create anything itself — it can't, it has no idea what "creating a volume" even means for localdir-csi specifically. It just picks up the phone and relays the request over gRPC: "make me a volume matching this card." Your CreateVolume method does the actual work and hands back a Volume, and external-provisioner turns that into a PersistentVolume object — a receipt, visible to the rest of the cluster, tying the original request card to a real, now-existing volume.

PersistentVolumeClaim objects, and external-provisioner actually watching one, are Chapter 8's territory; this chapter proves CreateVolume and DeleteVolume work the same way Chapter 6 proved NodePublishVolume did — with grpcurl, calling the RPC directly, no claim or clerk involved yet.

Driver gains a third embedded interface

ControllerServer is the third of the three interfaces Driver satisfies, alongside IdentityServer and NodeServer — same shape, same reason: a real csi.ControllerServer requires an unexported mustEmbedUnimplementedControllerServer() method, satisfiable only by embedding csi.UnimplementedControllerServer, for the same forward- compatibility reason Chapter 3 walked through for Identity. Update internal/driver/driver.go:

type Driver struct {
	csi.UnimplementedIdentityServer
	csi.UnimplementedControllerServer
	csi.UnimplementedNodeServer

	name    string
	version string
	health  HealthChecker
	nodeID  string
	dataDir string
	mount   Mounter
}

And add a third compile-time check alongside the existing two:

var _ csi.IdentityServer = (*Driver)(nil)
var _ csi.ControllerServer = (*Driver)(nil)
var _ csi.NodeServer = (*Driver)(nil)

Run the tests:

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Still green — and worth noticing what's different from every previous chapter's version of this same step. Chapters 5 and 6 both grew NewDriver's parameter list at the same time they added an interface, which broke every existing call site and meant a whole extra red/green cycle just to catch up. This time NewDriver doesn't change at all: Controller needs exactly two things Driver already has — dataDir, sitting there since Chapter 6, and nothing else. CreateVolume and DeleteVolume don't touch mount, don't touch health, don't need a fourth constructor parameter. Giving dataDir a home on Driver itself back in Chapter 6, rather than tucking it away inside Node's own state, is what makes it free to reach for here.

ControllerGetCapabilities, and why it can't stay empty

NodeGetCapabilities has returned an empty list since Chapter 5, and nothing has ever complained — kubelet doesn't gate anything on it. ControllerGetCapabilities is different, and skipping it isn't an option: external-provisioner calls it before it will call CreateVolume at all, and if the response doesn't list CREATE_DELETE_VOLUME among the driver's capabilities, external-provisioner treats that as "this driver doesn't do provisioning" and never sends a single CreateVolume request, no matter how many PersistentVolumeClaims show up. An empty list here wouldn't be a smaller, more honest answer the way it was for Node — it would silently disable the entire chapter.

Test first:

func TestControllerGetCapabilities(t *testing.T) {
	d := newTestDriver(t, dataDir, nil)

	resp, err := d.ControllerGetCapabilities(t.Context(), &csi.ControllerGetCapabilitiesRequest{})
	if err != nil {
		t.Fatalf("ControllerGetCapabilities returned an error: %v", err)
	}

	if len(resp.Capabilities) != 1 {
		t.Fatalf("got %d capabilities, want 1", len(resp.Capabilities))
	}

	rpc := resp.Capabilities[0].GetRpc()
	if rpc == nil {
		t.Fatal("expected an RPC capability, got something else")
	}
	if rpc.Type != csi.ControllerServiceCapability_RPC_CREATE_DELETE_VOLUME {
		t.Errorf("capability type = %v, want CREATE_DELETE_VOLUME", rpc.Type)
	}
}

Add this to a new file, internal/driver/controller_test.go, with "testing" and the csi package as its only imports for now. newTestDriver and dataDir need nothing new here — they're the same helper and the same package-level test constant testing_test.go and identity_test.go defined back in Chapter 6, and every file in the driver package shares them for free.

ControllerServiceCapability mirrors the shape PluginCapability used back in Chapter 3 for GetPluginCapabilities — a wrapper type with a oneof-style Type field, here holding an Rpc variant instead of a Service one, and a nested enum naming which specific capability this entry describes. GetRpc() is that variant's accessor, the same pattern GetService() was — note the capitalization: the underlying spec field is named rpc, and the generated Go code capitalizes field names letter-by-letter rather than treating rpc as an acronym, so it comes out Rpc, not RPC. The type nested one level down, RPC, keeps its own capitalization exactly as declared, since it's a message name, not a field name — this asymmetry (GetRpc() returning a *ControllerServiceCapability_RPC) is easy to mistype from memory and worth double-checking against the real generated code rather than guessing.

Run it:

go test ./internal/driver/...

Red, the familiar shape for a method still resolving to its embedded stub:

--- FAIL: TestControllerGetCapabilities (0.00s)
    controller_test.go:14: ControllerGetCapabilities returned an error: rpc error: code = Unimplemented desc = method ControllerGetCapabilities not implemented
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.002s
FAIL

Implement it. Create internal/driver/controller.go:

package driver

import (
	"context"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func (d *Driver) ControllerGetCapabilities(
	ctx context.Context,
	req *csi.ControllerGetCapabilitiesRequest,
) (*csi.ControllerGetCapabilitiesResponse, error) {
	return &csi.ControllerGetCapabilitiesResponse{
		Capabilities: []*csi.ControllerServiceCapability{
			{
				Type: &csi.ControllerServiceCapability_Rpc{
					Rpc: &csi.ControllerServiceCapability_RPC{
						Type: csi.ControllerServiceCapability_RPC_CREATE_DELETE_VOLUME,
					},
				},
			},
		},
	}, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green. The same Open/Closed shape Chapter 3 pointed out for GetPluginCapabilities applies again here: this returns a one-entry slice literal, not a hand-built response with a single hardcoded field, specifically so that a later chapter adding LIST_VOLUMES or GET_CAPACITY support means appending an entry, not restructuring this method or touching TestControllerGetCapabilities, which only ever asserts Capabilities[0].

CreateVolume, test-first: validation

CreateVolumeRequest carries more fields than this driver uses — secrets, volume_content_source, accessibility_requirements, and a newer mutable_parameters field the spec added after parameters already existed. Only two are things CreateVolume actually needs to check before doing anything: name, because every volume this driver creates is identified by it, and volume_capabilities, because the spec requires at least one entry describing how the volume needs to be usable, the same VolumeCapability shape NodePublishVolume already validates. One test at a time, same rhythm as every RPC so far:

func TestCreateVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.CreateVolumeRequest]{
		{
			name: "missing name",
			req: &csi.CreateVolumeRequest{
				VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
			},
		},
		{
			name: "missing volume capabilities",
			req: &csi.CreateVolumeRequest{
				Name: "vol-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.CreateVolumeRequest) error {
			_, err := d.CreateVolume(t.Context(), req)
			return err
		},
	)
}

No new imports needed. validationCase[T] and runValidation are the same generic helpers node_test.go reached for in Chapter 6 to collapse TestNodePublishVolume_Validation and TestNodeUnpublishVolume_Validation into a table plus one call — CreateVolume's own two-row validation table is exactly the shape they were built for, no new machinery required. mountCapability() is the same test-only helper Chapter 6 moved into testing_test.go — it lives in the same driver package, so every test file shares it without any new code.

Run it:

go test ./internal/driver/...

--- FAIL: TestCreateVolume_Validation (0.00s)
    --- FAIL: TestCreateVolume_Validation/missing_name (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{Name: CapacityRange:<nil> VolumeCapabilities:[0xc0000d00a8] Parameters:map[] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil>})
    --- FAIL: TestCreateVolume_Validation/missing_volume_capabilities (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{Name:vol-1 CapacityRange:<nil> VolumeCapabilities:[] Parameters:map[] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil>})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.005s
FAIL

Both failures point at testing_test.go:85, not at this test's own call to runValidation — the same t.Helper() mechanic Chapter 6 worked through for TestNodePublishVolume_Validation. requireStatusCode calls t.Helper(), so a failure inside it is attributed to whichever frame called it that isn't itself marked as a helper. That frame is the anonymous closure t.Run invokes inside runValidation — runValidation itself calls t.Helper() too, but the closure it defines does not — so the blame lands on testing_test.go:85, the requireStatusCode(...) line inside that closure, every time this pattern is used, in this chapter and every one after it.

Write just enough of CreateVolume to fix it — the two checks, and, for now, a stub success return. Create internal/driver/controller.go's new method (below ControllerGetCapabilities, with codes and status added to the file's imports):

func (d *Driver) CreateVolume(
	ctx context.Context,
	req *csi.CreateVolumeRequest,
) (*csi.CreateVolumeResponse, error) {
	if req.GetName() == "" {
		return nil, status.Error(codes.InvalidArgument, "name is required")
	}
	if len(req.GetVolumeCapabilities()) == 0 {
		return nil, status.Error(codes.InvalidArgument, "volume_capabilities is required")
	}

	return &csi.CreateVolumeResponse{Volume: &csi.Volume{}}, nil
}

That stub return is deliberately not nil, nil. CreateVolumeResponse's volume field is REQUIRED by the spec — a real CO receiving a nil Volume on an otherwise-successful response would be looking at a broken driver — so even a placeholder has to be a real, empty *Volume, never a missing one. It costs nothing here and avoids a self-inflicted nil-pointer panic the moment the next test reads a field off it.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green.

CreateVolume, test-first: actually creating something

Next case: a valid request should create a directory and hand back a Volume describing it.

func TestCreateVolume_CreatesTheVolume(t *testing.T) {
	dataDir := t.TempDir()
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateVolumeRequest{
		Name:               "vol-1",
		CapacityRange:      &csi.CapacityRange{RequiredBytes: 1 << 20},
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}

	resp, err := d.CreateVolume(t.Context(), req)
	if err != nil {
		t.Fatalf("CreateVolume() returned an error: %v", err)
	}

	if resp.Volume.VolumeId != "vol-1" {
		t.Errorf("VolumeId = %q, want %q", resp.Volume.VolumeId, "vol-1")
	}
	if resp.Volume.CapacityBytes != 1<<20 {
		t.Errorf("CapacityBytes = %d, want %d", resp.Volume.CapacityBytes, 1<<20)
	}

	if info, err := os.Stat(filepath.Join(dataDir, "vol-1")); err != nil || !info.IsDir() {
		t.Fatalf("volume directory was not created: %v", err)
	}
}

Add "os" and "path/filepath" to controller_test.go's imports.

go test ./internal/driver/...

Red — the previous step's stub returns an empty Volume and creates nothing:

--- FAIL: TestCreateVolume_CreatesTheVolume (0.00s)
    controller_test.go:71: VolumeId = "", want "vol-1"
    controller_test.go:74: CapacityBytes = 0, want 1048576
    controller_test.go:78: volume directory was not created: stat /tmp/TestCreateVolume_CreatesTheVolume.../vol-1: no such file or directory
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

Fix it. This is the same filepath.Join(d.dataDir, ...) convention NodePublishVolume already assumes in Chapter 6 — a volume's data lives in a directory named after its own ID, directly under the driver's data root. CreateVolume is what actually creates that directory, using a volume's name as its ID:

	path := filepath.Join(d.dataDir, req.GetName())
	if err := os.MkdirAll(path, 0o750); err != nil {
		return nil, status.Errorf(codes.Internal, "creating volume %q: %v", req.GetName(), err)
	}

	return &csi.CreateVolumeResponse{
		Volume: &csi.Volume{
			VolumeId:      req.GetName(),
			CapacityBytes: req.GetCapacityRange().GetRequiredBytes(),
		},
	}, nil

That replaces the old stub return — insert it right after the two validation checks, and add "os" and "path/filepath" to controller.go's imports too.

req.GetCapacityRange().GetRequiredBytes() chains two nil-safe accessors on purpose: capacity_range is an OPTIONAL field of CreateVolumeRequest, so a real request might not set it at all, and calling GetRequiredBytes() on a nil *CapacityRange needs to return 0 rather than panic — the same nil-receiver pattern every generated getter in the csi package already follows, req.GetVolumeId() included.

Worth being honest about what this line does and doesn't do: localdir-csi never actually enforces CapacityRange — nothing checks that the underlying disk has RequiredBytes free, and nothing prevents someone writing more data into the volume than they asked for. It just remembers whatever number it was given and reports it back, which is enough to satisfy a PersistentVolumeClaim's status.capacity in Chapter 8, but a real backend — one that actually provisions block devices or files of a specific size — is where that number would turn into an actual constraint. RequiredBytes and LimitBytes sitting at 0 (their zero value, if a CO never sets CapacityRange at all) is commonly treated as "no particular size requested" by drivers in practice, but that's a convention this codebase is choosing to follow, not something the spec text itself states as a MUST.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green.

CreateVolume, test-first: idempotency

The spec is specific about what CreateVolume has to do when it's asked to create a volume that already exists: MUST reply 0 OK if the existing volume matches what was asked for, and there's a good reason — PersistentVolumeClaim provisioning isn't guaranteed exactly-once. external-provisioner can retry a CreateVolume call it never got a response for, and a driver that treats the second call as an error turns a harmless network hiccup into a permanently stuck claim.

func TestCreateVolume_Idempotent(t *testing.T) {
	dataDir := t.TempDir()
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateVolumeRequest{
		Name:               "vol-1",
		CapacityRange:      &csi.CapacityRange{RequiredBytes: 1 << 20},
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}

	if _, err := d.CreateVolume(t.Context(), req); err != nil {
		t.Fatalf("first CreateVolume() returned an error: %v", err)
	}

	resp, err := d.CreateVolume(t.Context(), req)
	if err != nil {
		t.Fatalf("second CreateVolume() for the same name returned an error: %v", err)
	}
	if resp.Volume.VolumeId != "vol-1" {
		t.Errorf("VolumeId = %q, want %q", resp.Volume.VolumeId, "vol-1")
	}
}

Run it:

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Already green, no implementation change needed — the same shape of free correctness Chapter 6 found in NodeUnpublishVolume, just on the other side of the volume's lifecycle. os.MkdirAll doesn't error when the directory it's asked to create already exists; it only errors when something actually goes wrong. Calling CreateVolume twice for the same name calls MkdirAll twice on the same path, and the second call is a no-op success, not a repeat write, not a conflict.

That's real idempotency, but it's a narrower version than the spec actually asks for. The full requirement is to reply OK only when the existing volume matches the new request's capacity, capabilities, and parameters, and reply ALREADY_EXISTS (gRPC code 6) when the name matches but something else about the request doesn't. This driver has no way to ask that second question yet — a bare directory on disk doesn't remember what capacity or parameters it was created with, only that it exists.

Chapter 6 already faced this exact question for NodePublishVolume, and answered it with a small sidecar metadata file: record what a call actually did, and compare a later call against that record instead of trusting a side effect alone. The same idea would close this gap here, too. It isn't applied yet, on purpose — Chapter 9 points a real spec-conformance tool, csi-sanity, straight at this exact gap, and it's worth watching a real tool catch a real gap before reaching for the fix on suspicion alone. CreateVolume gets its own metadata file in Chapter 9, once that confirmation is in hand. Worth knowing the gap is there in the meantime, even though nothing in this book's test suite exercises it yet.

DeleteVolume, test-first

Same three-cycle rhythm as NodeUnpublishVolume in Chapter 6: validation, the actual removal, then idempotency. DeleteVolumeRequest only has two fields, volume_id and secrets, and only volume_id needs checking.

func TestDeleteVolume_Validation(t *testing.T) {
	d := newTestDriverInTempDir(t)

	_, err := d.DeleteVolume(t.Context(), &csi.DeleteVolumeRequest{})
	requireStatusCode(t, err, codes.InvalidArgument, &csi.DeleteVolumeRequest{})
}

Add "google.golang.org/grpc/codes" to controller_test.go's imports — requireStatusCode itself wraps status.Code internally, so nothing here needs "google.golang.org/grpc/status" directly, unlike the pre-refactor version of this test.

go test ./internal/driver/...

--- FAIL: TestDeleteVolume_Validation (0.00s)
    controller_test.go:110: code = Unimplemented, want InvalidArgument (req=&{VolumeId: Secrets:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

This one lands back on controller_test.go itself, not testing_test.go — a single direct call to requireStatusCode, not one routed through runValidation's closure, so the t.Helper() skip lands on this test's own line, exactly the way it did for the single-case TestNodePublishVolume_VolumeNotFound-style calls in Chapter 6.

func (d *Driver) DeleteVolume(
	ctx context.Context,
	req *csi.DeleteVolumeRequest,
) (*csi.DeleteVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}

	return &csi.DeleteVolumeResponse{}, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green. DeleteVolumeResponse is intentionally empty — unlike CreateVolumeResponse, the spec gives it no fields at all, so there's nothing a stub could get subtly wrong the way CreateVolume's nil Volume could.

Now the actual removal:

func TestDeleteVolume_RemovesTheVolume(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.DeleteVolumeRequest{VolumeId: "vol-1"}

	if _, err := d.DeleteVolume(t.Context(), req); err != nil {
		t.Fatalf("DeleteVolume() returned an error: %v", err)
	}

	if _, err := os.Stat(filepath.Join(dataDir, "vol-1")); !os.IsNotExist(err) {
		t.Errorf("volume directory still exists after DeleteVolume")
	}
}

makeVolumeDir is the same testing_test.go helper Chapter 6 introduced for TestNodePublishVolume_MountsTheVolume — a fake volume directory is a fake volume directory whether Node or Controller is the one about to act on it, so there's nothing Controller-specific to write here at all.

go test ./internal/driver/...

--- FAIL: TestDeleteVolume_RemovesTheVolume (0.00s)
    controller_test.go:125: volume directory still exists after DeleteVolume
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

	path := filepath.Join(d.dataDir, req.GetVolumeId())
	if err := os.RemoveAll(path); err != nil {
		return nil, status.Errorf(codes.Internal, "deleting volume %q: %v", req.GetVolumeId(), err)
	}

	return &csi.DeleteVolumeResponse{}, nil

That replaces DeleteVolume's old stub return, right after the validation check.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green. Last case:

func TestDeleteVolume_Idempotent(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.DeleteVolumeRequest{VolumeId: "does-not-exist"}

	if _, err := d.DeleteVolume(t.Context(), req); err != nil {
		t.Fatalf("DeleteVolume() for an already-gone volume returned an error: %v", err)
	}
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Also already green, and it's the exact same lesson NodeUnpublishVolume taught in Chapter 6, on the opposite side of a volume's lifecycle: os.RemoveAll doesn't error when the path it's asked to remove doesn't exist. The spec's requirement here is actually simpler than CreateVolume's — MUST reply 0 OK if the volume doesn't exist, full stop, no capacity or parameter comparison to get right — and this driver gets it for free from the exact same standard-library behavior that made deleting an already-unmounted target free in Chapter 6.

Wiring ControllerServer into main.go

One line, alongside the existing RegisterIdentityServer and RegisterNodeServer calls:

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	csi.RegisterControllerServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

Nothing else in main.go changes. The same d — one Driver, one *mount.New(""), one dataDir — satisfies all three services, and this line is the entire cost of exposing a third one over the same socket. It's worth pausing on that "same socket" — this chapter's Controller Deployment and Chapter 4/5/6's Node DaemonSet both run this exact same binary, unmodified, main.go and all. What differs between them isn't the code; it's which container each one runs in, and which sidecar sits next to it. A real, larger CSI driver sometimes splits Controller and Node into genuinely separate binaries, often to keep a Controller image free of node-local dependencies a Deployment will never need — but nothing about the CSI spec requires that split, and localdir-csi doesn't bother with it.

deploy/controller.yaml

Chapter 2's project layout sketched this file as "driver + provisioner + attacher sidecars" — a reasonable guess at the time, but only partly right, as Chapter 8 explains once ControllerPublishVolume exists to look at. deploy/csidriver.yaml already set attachRequired: false in Chapter 4, and Chapter 8, not this one, is where ControllerPublishVolume gets implemented. For now, controller.yaml needs exactly one sidecar: external-provisioner, the component that turns PersistentVolumeClaims into CreateVolume calls.

Create deploy/controller.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: localdir-csi-controller
  labels:
    app: localdir-csi-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: localdir-csi-controller
  template:
    metadata:
      labels:
        app: localdir-csi-controller
    spec:
      serviceAccountName: localdir-csi-controller
      containers:
        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          args:
            - -endpoint=unix:///csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
        - name: external-provisioner
          image: registry.k8s.io/sig-storage/csi-provisioner:v6.3.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
      volumes:
        - name: socket-dir
          emptyDir: {}
        - name: data-dir
          hostPath:
            path: /var/lib/localdir-csi/data
            type: DirectoryOrCreate

A few things that are different from node.yaml, on purpose:

socket-dir is an emptyDir, not a hostPath. Chapter 4's DaemonSet needed hostPath specifically so kubelet — a process running directly on the node, entirely outside Kubernetes — could reach the socket too. Nothing outside this pod ever needs to reach the Controller's socket; external-provisioner is a container in the same pod, so a socket that only has to be visible to two containers sharing one pod is exactly what emptyDir is for, no host filesystem involved at all.

data-dir still uses hostPath, at the identical path /var/lib/localdir-csi/data the Node DaemonSet mounts. That's not incidental — CreateVolume and NodePublishVolume both compute a volume's directory as filepath.Join(dataDir, volumeID), and for NodePublishVolume to find a directory CreateVolume created, both have to agree on where dataDir actually is on disk. On the single-node kind cluster this book deploys to, that's automatic — Controller and every Node pod land on the one same machine, so the same host path is the same directory no matter which pod wrote to it. On a real multi-node cluster, that assumption breaks completely: a Controller pod scheduled on node A creating a directory has done nothing for a pod trying to publish that volume from node B, since /var/lib/localdir-csi/data on two different nodes are two entirely different directories. This is localdir-csi's central, load-bearing simplification, not a small detail — it's why the driver's name starts with "local." A real network-backed driver, EBS or NFS or anything else with data reachable from more than one node, doesn't have this problem, because its CreateVolume doesn't write to any one node's local disk to begin with.

No --leader-election flag. external-provisioner supports running multiple replicas with one active leader at a time, coordinated through Kubernetes Lease objects, for driver deployments that want high availability. That needs its own RBAC rule and its own flag, and neither is worth adding for a single-replica Deployment where there's only ever one instance to begin with — a lock with no contender to lock out.

No privileged, no mountpoint-dir. Everything node.yaml's securityContext and extra hostPath volumes exist for — performing a real bind mount, seeing kubelet's own pod directories — is Node-service territory. CreateVolume and DeleteVolume only ever call os.MkdirAll and os.RemoveAll against an ordinary mounted directory; there's nothing here that needs elevated privileges.

RBAC: the first Kubernetes-API-calling container in this book

Every container deployed so far — localdir-csi itself, and node-driver-registrar — has never made a single call to the Kubernetes API. node-driver-registrar only ever talks to a local socket and to kubelet, a process on the same machine, neither of which needs Kubernetes credentials. external-provisioner is different: watching PersistentVolumeClaims cluster-wide, and creating PersistentVolumes in response, both go through the real Kubernetes API server, which means external-provisioner needs to authenticate as something, and that something needs specific permission to read and write specific object types.

A ServiceAccount is that "something" — an identity a pod can run as, distinct from any human user's own credentials. A ClusterRole lists what actions are allowed on which resource types, cluster-wide rather than scoped to one namespace, since PersistentVolumeClaims can live in any namespace. A ClusterRoleBinding connects the two — "this ServiceAccount gets these permissions" — because a ClusterRole on its own grants nothing to anyone until something binds it to an identity.

Create deploy/controller-rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: localdir-csi-controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: localdir-csi-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "patch", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: localdir-csi-provisioner
subjects:
  - kind: ServiceAccount
    name: localdir-csi-controller
    namespace: default
roleRef:
  kind: ClusterRole
  name: localdir-csi-provisioner
  apiGroup: rbac.authorization.k8s.io

This is a trimmed version of external-provisioner's own published RBAC rules — real CSI sidecars ship a reference manifest listing exactly what they need, and the honest way to write this file is to start from that reference and remove what a specific driver's feature set doesn't use, not to guess. Two categories are already gone here on purpose: snapshot-related rules (volumesnapshots, volumesnapshotcontents) are absent because localdir-csi doesn't implement CreateSnapshot, and volumeattachments is absent because that resource only matters for drivers with the PUBLISH_UNPUBLISH_VOLUME capability — the one ControllerGetCapabilities deliberately doesn't list, since ControllerPublishVolume isn't implemented until Chapter 8. Namespace- scoped rules for leases and csistoragecapacities are also absent, for the same reason --leader-election is absent from controller.yaml: nothing here uses either feature.

patch on persistentvolumes earns its own callout, since it's easy to trim by mistake — a plain create/delete pair looks complete, but isn't. Modern external-provisioner adds a finalizer to every PV it creates, and only removes that finalizer once DeleteVolume has actually succeeded. Removing a finalizer is a small, partial edit to an existing object — exactly what patch is for, not update, which would mean sending the whole object back. Without patch, external-provisioner can still create and delete volumes just fine, right up until a PersistentVolumeClaim actually gets deleted — at that point it gets stuck retrying forever, unable to clear its own finalizer, and the real symptom is an RBAC "forbidden" error naming patch specifically, not delete. persistentvolumeclaims needs update for a related reason: to write back its own status on the claim as provisioning progresses, not to delete anything.

namespace: default in the ClusterRoleBinding's subjects matters: it has to match whatever namespace controller.yaml's Deployment actually runs in — the ServiceAccount a pod uses is namespaced, even though the permissions it's bound to, via ClusterRole, are not.

Deploying

make deploy

kubectl get pods -l app=localdir-csi-controller

NAME                                        READY   STATUS    RESTARTS   AGE
localdir-csi-controller-6f7d8b9c4d-x7k2p    2/2     Running   0          14s

2/2, the same signal Chapter 4 first explained — two containers, localdir-csi and external-provisioner, both up. Check what external-provisioner itself thinks of the driver it's sitting next to:

kubectl logs -l app=localdir-csi-controller -c external-provisioner --tail=20

I0820 12:00:01.442851       1 csi-provisioner.go:159] Version: v6.3.0
I0820 12:00:01.443988       1 connection.go:246] Connecting to unix:///csi/csi.sock
I0820 12:00:01.612300       1 common.go:143] Probing CSI driver for readiness
I0820 12:00:01.615117       1 csi-provisioner.go:222] Detected CSI driver localdir.csi.example.com
I0820 12:00:01.615204       1 controller.go:833] Starting provisioner controller localdir.csi.example.com_localdir-csi-controller-6f7d8b9c4d-x7k2p_...

Detected CSI driver localdir.csi.example.com is external-provisioner successfully calling GetPluginInfo, from Chapter 3, over the shared socket. Nothing about CREATE_DELETE_VOLUME shows up explicitly in this log — it's checked silently, and its absence would show up not as an error here, but as external-provisioner simply never attempting a CreateVolume call later, the same "quiet failure" shape Chapter 4 already warned --kubelet-registration-path mistakes produce.

Proving it, with grpcurl

Same tool as Chapter 6, but it has to reach the driver a different way this time. Chapter 6's grpcurl lived on the kind node itself, because the Node service's socket sat on a hostPath — a real directory on that same node's disk. The Controller's socket is deliberately an emptyDir instead, visible only to containers inside its own pod, which means grpcurl needs to run inside that pod, not on the node next to it.

kubectl cp copies a file straight into a running container, using the exact same grpcurl-linux binary Chapter 6 already pulled out of grpcurl's official image:

kubectl cp ./grpcurl-linux \
  $(kubectl get pod -l app=localdir-csi-controller -o jsonpath='{.items[0].metadata.name}'):/tmp/grpcurl \
  -c localdir-csi
kubectl exec deploy/localdir-csi-controller -c localdir-csi -- chmod +x /tmp/grpcurl

The $(kubectl get pod ...) substitution is only there because kubectl cp, unlike kubectl exec, doesn't understand a deploy/name shorthand — it needs one specific pod's actual name, so this looks it up by the same app label the Deployment's own pod template sets.

Now call it, with -c localdir-csi naming which of the pod's two containers to exec into, the same way logs -c external-provisioner did above:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "test-vol-1",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

{
  "volume": {
    "capacityBytes": "1048576",
    "volumeId": "test-vol-1"
  }
}

The same volume ID as the data-dir bind mount from Chapter 6's hello.txt proof, but arriving from the opposite direction — this time, localdir-csi created the directory, instead of assuming it already existed:

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/

test-vol-1

Call it again — the exact same request, unchanged:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "test-vol-1",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

{
  "volume": {
    "capacityBytes": "1048576",
    "volumeId": "test-vol-1"
  }
}

Identical response, 0 OK, not an error — the idempotency TestCreateVolume_Idempotent proved with a fake dataDir now proved again for real, against an actual gRPC socket in an actual kind node.

Now delete it:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "test-vol-1"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteVolume

{}

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/

Empty — gone. And once more, against a volume ID that's already gone:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "test-vol-1"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteVolume

{}

Still 0 OK — TestDeleteVolume_Idempotent's claim, proved the same way.

What you should have now

• internal/driver/driver.go: Driver now embeds all three Unimplemented*Server stubs and satisfies all three CSI service interfaces, with no change to NewDriver's signature at all
• internal/driver/controller.go: ControllerGetCapabilities advertising CREATE_DELETE_VOLUME, and CreateVolume/DeleteVolume, both idempotent — one by explicit design, one for free from os.MkdirAll and os.RemoveAll's own behavior
• cmd/localdir-csi/main.go registering csi.ControllerServer over the same socket and the same Driver value as Identity and Node
• deploy/controller.yaml: a single-replica Deployment running localdir-csi alongside external-provisioner, sharing a socket over emptyDir rather than hostPath, and mounting the same data-dir every Node pod does
• deploy/controller-rbac.yaml: the first ServiceAccount, ClusterRole, and ClusterRoleBinding this book has needed, scoped to exactly what external-provisioner's actual feature set here requires
• Direct, hands-on proof — grpcurl calls against a real Controller pod — that CreateVolume and DeleteVolume both work, and that calling either one twice is safe

Nothing in this chapter has been triggered by an actual PersistentVolumeClaim yet — every call so far has been grpcurl, typed by hand, the same bridge Chapter 6 used before a real pod ever requested a volume. Chapter 8 closes that gap: a PersistentVolumeClaim that external-provisioner notices on its own, a PersistentVolume it creates in response, and a pod that mounts it — the whole pipeline, end to end, with nothing typed into grpcurl at all.

Chapter 8: Attaching Volumes

Chapter 7 gave the Controller service a real CreateVolume and DeleteVolume, proved with grpcurl typed by hand against a real pod. Two things are still missing before this driver behaves the way a real Kubernetes user would actually experience it. First, there's a whole category of the CSI spec this book hasn't touched yet — "attaching" a volume — and it's worth understanding what that means and whether localdir-csi needs it, rather than skipping it silently. Second, and bigger: nothing so far has ever been triggered by Kubernetes itself. Every RPC call in this book has been typed into grpcurl by a person. This chapter closes both gaps.

What "attaching" a volume actually means

Picture a USB drive. Right now it's sitting on your desk, unplugged.

The drive already has data on it. It already has a filesystem. It already has a fixed capacity. None of that depends on being plugged in.

But your laptop can't see any of it yet. It can't mount the drive. It can't read a single file off it. Not because the drive is broken — just because the two aren't connected yet.

Now plug it in. Nothing on the drive changes. No new files appear. The data was always there. What changes is simpler than that: your laptop can suddenly see the drive. That's the whole effect of plugging it in — reachability, not creation.

Unplug it again. Carry it to a different laptop. Plug it in there instead. That laptop can see it now. Usually only one laptop at a time, though — plugging a drive into a second machine while it's still in the first one doesn't work the way you'd want it to.

That plug-in step — making something that already exists reachable from one specific machine — is exactly what ControllerPublishVolume does in CSI. Say it again, plainly: publishing a volume doesn't create it. Publishing a volume just makes it reachable from one node.

Here's where that matters for real. Take an AWS EBS volume. It lives on Amazon's own storage infrastructure, not on any one of your nodes. Before a pod can mount it, someone has to tell AWS: "attach volume vol-123 to instance i-0abc." That one API call is the cloud version of plugging in the USB drive. Skip it, and there's nothing for the node to mount — the same way an unplugged USB drive has nothing for your laptop to mount.

Now back to localdir-csi. Its volumes were never unplugged from anywhere. There was never a separate drive to begin with.

CreateVolume just makes a directory. That directory lives at /var/lib/localdir-csi/data/<volume-id>, right there on the node's own local disk. Chapter 7 already explained why that exact path has to match on every node in this cluster — same reasoning, same path, still true here.

There's no second piece of storage sitting off to the side, waiting to be attached. The "drive," if you want to keep calling it that, was built into the machine from day one. It's like your laptop's own internal disk: you don't plug that in before saving a file to it. It's just already there.

So, one more time, plainly: this driver has nothing to attach, because it never had anything separate to begin with.

That still leaves a real question for this chapter, though. CSI's spec defines ControllerPublishVolume and ControllerUnpublishVolume whether or not a given driver's storage needs them — the two methods exist in the interface either way. So the question isn't "can we skip writing these methods." It's "should this driver use them the same way a driver with a real attach step would." And the honest answer is no.

Implementing them anyway, test-first

Spec-legal or not, ControllerPublishVolume and ControllerUnpublishVolume are still two ordinary RPCs with defined request and response shapes, and building them the same test-first way as every other RPC in this book is worth doing — both for the practice, and because a later chapter (csi-sanity, Chapter 9) will call every method a ControllerServer exposes, implemented or not.

ControllerPublishVolumeRequest carries volume_id, node_id, and volume_capability, all three marked REQUIRED by the spec. readonly is also REQUIRED as a field. But it's a plain bool with a usable zero value. There's nothing to validate there the way there is for a string that might be empty, or a pointer that might be nil.

The spec text adds one more rule for node_id: "the CO SHALL set this field to match the node ID returned by NodeGetInfo." Read that rule carefully, because it's easy to misread. It's a promise the CO makes. It's not a check the SP runs. ControllerPublishVolume below does not, and will not, verify that promise — it only checks that node_id isn't empty, the same shallow check every other required string here gets.

That's worth saying plainly, not glossing over: this driver trusts the CO's node_id, it doesn't verify it. And there's a concrete reason it can't easily do better. d.nodeID — the field NodeGetInfo reads from, back in Chapter 5 — only gets set correctly on the Node DaemonSet, where deploy/node.yaml wires up the NODE_NAME Downward API field. The Controller Deployment doesn't set that same env var. So on the Controller pod, Chapter 5's own fallback path kicks in instead — d.nodeID ends up as the pod's own hostname, something like localdir-csi-controller-6f7d8b9c4d-x7k2p, not a real cluster node name. Chapter 5 already explained exactly why that value is wrong for this purpose. Comparing req.GetNodeId() against a value that's already known to be wrong wouldn't add a real check — it would just add a check that's guaranteed to fail. So ControllerPublishVolume doesn't attempt it, and says so honestly instead of pretending to validate something it can't.

It's tempting to add that comparison anyway. Something like this:

if req.GetNodeId() != d.nodeID {
	return nil, status.Errorf(codes.InvalidArgument,
		"driver nodeID %v does not match request nodeID %v", d.nodeID, req.GetNodeId())
}

Don't. Two separate problems, not one.

The first is the concrete one, from just above: on the real Controller pod, d.nodeID isn't a node name. It's a pod hostname. This check would compare a real, correct node_id against a value already known to be wrong. Every publish call would fail — including the exact grpcurl call this chapter runs a few pages from now.

The second is deeper, and worth understanding even outside this one bug. One Controller pod serves the whole cluster, not one node. It has to accept node_id values naming any node out there — not just whichever node it happens to be scheduled on itself. d.nodeID answers a different question: "which node is this process running on." That question only makes sense for the Node service, where one pod really does correspond to one node. Reusing the same field on the Controller side quietly asks it something it was never built to answer. Fixing the pod-hostname issue wouldn't fix this — even a perfectly correct d.nodeID on the Controller side would still only ever equal one node, never many.

A real version of this check would need a real list of known nodes to check against — something like reading Kubernetes's own Node or CSINode objects through the API, which this driver's ServiceAccount already has read access to (Chapter 7's RBAC includes nodes, for external-provisioner's own use) but which ControllerPublishVolume itself never calls. Worth knowing, too: even a driver that does run this check should return NOT_FOUND (per the spec's own error table), not InvalidArgument — a well-formed node_id naming a node that doesn't exist isn't a malformed request, it's a request about something missing.

func TestControllerPublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.ControllerPublishVolumeRequest]{
		{
			name: "missing volume id",
			req: &csi.ControllerPublishVolumeRequest{
				NodeId:           "test-node-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing node id",
			req: &csi.ControllerPublishVolumeRequest{
				VolumeId:         "vol-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing volume capability",
			req: &csi.ControllerPublishVolumeRequest{
				VolumeId: "vol-1",
				NodeId:   "test-node-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.ControllerPublishVolumeRequest) error {
			_, err := d.ControllerPublishVolume(t.Context(), req)
			return err
		},
	)
}

Driver already embeds UnimplementedControllerServer, same as it has since Chapter 7, and that embedded type already has stub ControllerPublishVolume/ControllerUnpublishVolume methods returning Unimplemented — so, unlike Chapter 5 or 7, there's no interface to add and no NewDriver signature to touch this time. No new imports either: validationCase[T] and runValidation are the same generic helpers Chapters 6 and 7 already reached for, and this three-row table is exactly the shape they exist to collapse. The method just needs a real body:

--- FAIL: TestControllerPublishVolume_Validation (0.00s)
    --- FAIL: TestControllerPublishVolume_Validation/missing_volume_id (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: NodeId:test-node-1 VolumeCapability:0xc00000e300 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestControllerPublishVolume_Validation/missing_node_id (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 NodeId: VolumeCapability:0xc00000e318 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestControllerPublishVolume_Validation/missing_volume_capability (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 NodeId:test-node-1 VolumeCapability:<nil> Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

code = Unimplemented, exactly as the paragraph above predicted — that's the embedded stub talking, not this test's own validation. Still red, and still for the right reason: nothing here validates its arguments yet, it just happens to fail with a more specific code than "missing" would. And once again every failure lands on testing_test.go:85, not on this test's own call to runValidation — the same t.Helper() mechanic Chapters 6 and 7 already worked through, holding here without any new explanation needed.

func (d *Driver) ControllerPublishVolume(
	ctx context.Context,
	req *csi.ControllerPublishVolumeRequest,
) (*csi.ControllerPublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetNodeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "node_id is required")
	}
	if req.GetVolumeCapability() == nil {
		return nil, status.Error(codes.InvalidArgument, "volume_capability is required")
	}

	return &csi.ControllerPublishVolumeResponse{}, nil
}

Green — and now the interesting question: what should ControllerPublishVolume actually do, once its arguments pass validation? The spec's error-code table gives a direct answer: NOT_FOUND (5) if "the volume does not exist." A volume genuinely can be published against a volume ID nothing created — a stale claim, a typo, a race — so that's worth its own test, ahead of writing the happy path:

func TestControllerPublishVolume_VolumeNotFound(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "does-not-exist",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	_, err := d.ControllerPublishVolume(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

requireStatusCode takes whatever codes.Code a test expects, not just InvalidArgument — the same helper Chapter 7 wrote fits a NotFound assertion exactly as well as a validation one.

--- FAIL: TestControllerPublishVolume_VolumeNotFound (0.00s)
    controller_test.go:181: code = OK, want NotFound (req=&{VolumeId:does-not-exist NodeId:test-node-1 VolumeCapability:0xc00000e438 Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

This one lands on controller_test.go directly, not testing_test.go — a single direct call to requireStatusCode, the same shape as Chapter 7's TestDeleteVolume_Validation, not one routed through runValidation's closure.

Red, because right now ControllerPublishVolume doesn't check anything beyond its own arguments — it happily reports success for a volume that was never created. The fix reuses a pattern this book has leaned on since NodePublishVolume in Chapter 6: a volume's existence, as far as this driver is concerned, is whether filepath.Join(d.dataDir, volumeID) exists on disk.

	path := filepath.Join(d.dataDir, req.GetVolumeId())
	if _, err := os.Stat(path); os.IsNotExist(err) {
		return nil, status.Errorf(codes.NotFound, "volume %q not found", req.GetVolumeId())
	} else if err != nil {
		return nil, status.Errorf(codes.Internal, "checking volume %q: %v", req.GetVolumeId(), err)
	}

=== RUN   TestControllerPublishVolume_VolumeNotFound
--- PASS: TestControllerPublishVolume_VolumeNotFound (0.00s)

That os.Stat call runs inside the Controller pod's own container. Worth asking directly: whose disk is it actually checking?

In this book's cluster, there's only one possible answer. It's the one kind node. Controller and every Node pod are containers on that same one machine. They share the exact same hostPath, /var/lib/localdir-csi/data — Chapter 7 already pointed this out, when deploy/controller.yaml first mounted it. So this os.Stat check, and NodePublishVolume's own existence check later, in a completely different pod, end up looking at the exact same file on the exact same disk. It only looks like two separate checks.

A real multi-node cluster breaks that. Say the Controller pod lands on node A. CreateVolume, back in Chapter 7, creates the volume's directory on node A's own local disk — wherever Controller happened to be scheduled, not wherever the volume is actually needed. Now say a pod that wants this volume gets scheduled onto node B instead. NodePublishVolume runs inside a Node pod on node B. It checks node B's own local disk. Node A's directory was never there to begin with, and nothing in this driver ever copies data between nodes.

So this os.Stat check depends on the same fact Chapter 7 already named as this driver's "central, load-bearing simplification... why the driver's name starts with 'local.'" That fact doesn't just affect CreateVolume. It quietly reaches into every method that computes filepath.Join(dataDir, volumeID) and treats the result as meaningful — ControllerPublishVolume included. This is the same gap, seen again from a different angle.

Two more cases round this method out, and both go straight to green without any further production code — which is itself worth pausing on, rather than treating as nothing happening. A happy-path test, publishing a volume that genuinely exists:

func TestControllerPublishVolume_Succeeds(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "vol-1",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerPublishVolume() returned an error: %v", err)
	}
}

makeVolumeDir again, same as Chapter 7's TestDeleteVolume_RemovesTheVolume — a fake volume on disk is a fake volume on disk, regardless of which Controller method is about to look for it.

And an idempotency test, calling it twice for the same volume and node:

func TestControllerPublishVolume_Idempotent(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "vol-1",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("first ControllerPublishVolume() returned an error: %v", err)
	}
	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("second ControllerPublishVolume() for the same volume/node returned an error: %v", err)
	}
}

Both pass immediately. That's not a wasted test — it's the test proving a specific claim: that a directory-existence check has no state of its own to become inconsistent on a second call. CreateVolume's idempotency back in Chapter 7 needed os.MkdirAll's own already-exists-is-fine behavior; this one gets the same property for free from os.Stat never changing anything. Writing the test first either catches a case where that assumption is wrong, or — as here — turns "I'm pretty sure this is already idempotent" into "a test says so."

ControllerUnpublishVolume is smaller. Its request has only volume_id REQUIRED — node_id is explicitly OPTIONAL here, unlike its counterpart on the publish side, and the spec text explains why, using its own shorthand for "the driver itself," the SP (Storage Plugin) — the other half of the CO/SP pair Chapter 6 already introduced CO for: "If the value is set, the SP MUST unpublish the volume from the specified node. If the value is unset, the SP MUST unpublish the volume from all nodes it is published to." A single-node driver like this one never has to act on that distinction, but the validation has to reflect it correctly regardless — requiring node_id here would reject a spec-legal request.

func TestControllerUnpublishVolume_Validation(t *testing.T) {
	d := newTestDriverInTempDir(t)

	_, err := d.ControllerUnpublishVolume(t.Context(), &csi.ControllerUnpublishVolumeRequest{})
	requireStatusCode(t, err, codes.InvalidArgument, &csi.ControllerUnpublishVolumeRequest{})
}

--- FAIL: TestControllerUnpublishVolume_Validation (0.00s)
    controller_test.go:223: code = Unimplemented, want InvalidArgument (req=&{VolumeId: NodeId: Secrets:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.005s
FAIL

func (d *Driver) ControllerUnpublishVolume(
	ctx context.Context,
	req *csi.ControllerUnpublishVolumeRequest,
) (*csi.ControllerUnpublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}

	return &csi.ControllerUnpublishVolumeResponse{}, nil
}

The spec's idempotency text for unpublish is more forgiving than publish's, on purpose — it says a driver "SHOULD" return 0 OK even when the volume or node "can not be found and can be safely regarded as ControllerUnpublished," rather than the MUST NOT succeed silently tone ControllerPublishVolume takes toward a genuinely missing volume. That asymmetry exists for a real reason: a CO retrying an unpublish call after a partial failure needs "already gone" to look identical to "successfully removed," or cleanup logic ends up stuck re-requesting an unpublish that can never appear to succeed. This driver's implementation already matches that — it doesn't check the volume exists at all, so "volume already gone" and "volume never existed" both fall through to the same 0 OK — which the tests confirm directly, no new production code required for any of them:

func TestControllerUnpublishVolume_Succeeds(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "vol-1",
		NodeId:   "test-node-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() returned an error: %v", err)
	}
}

func TestControllerUnpublishVolume_NodeIDIsOptional(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "vol-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() with no node_id returned an error: %v", err)
	}
}

func TestControllerUnpublishVolume_Idempotent(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "already-unpublished",
		NodeId:   "test-node-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() for an already-unpublished volume returned an error: %v", err)
	}
}

=== RUN   TestControllerUnpublishVolume_Succeeds
--- PASS: TestControllerUnpublishVolume_Succeeds (0.00s)
=== RUN   TestControllerUnpublishVolume_NodeIDIsOptional
--- PASS: TestControllerUnpublishVolume_NodeIDIsOptional (0.00s)
=== RUN   TestControllerUnpublishVolume_Idempotent
--- PASS: TestControllerUnpublishVolume_Idempotent (0.00s)
PASS

The choice not to advertise any of this

Both methods work, are tested, and would behave correctly if a CO called them. None of that means Kubernetes should ever be told to call them — and here the earlier "USB drive that was never unplugged" framing turns into a concrete set of decisions, each one worth making on purpose rather than by accident.

ControllerGetCapabilities, from Chapter 7, still reports only CREATE_DELETE_VOLUME. It is not getting PUBLISH_UNPUBLISH_VOLUME added now that the RPC exists — the capability list describes what a CO should rely on this driver for, not what methods happen to have a non-stub body. Advertising a capability a driver has no real use for would be actively misleading, not just unnecessary.

That decision connects to a piece of Kubernetes machinery this book hasn't described yet: VolumeAttachment objects. When a CSI driver does need attach/detach, Kubernetes's own in-tree attach/detach controller — not any sidecar container — is what creates a VolumeAttachment object for a volume about to be mounted, and it's external-attacher's job to notice that object and turn it into the actual ControllerPublishVolume gRPC call. Critically, that controller only creates VolumeAttachment objects at all when the driver's own CSIDriver object has spec.attachRequired: true — and Chapter 4 already set attachRequired: false in deploy/csidriver.yaml, for exactly this driver. With that field false, no VolumeAttachment object is ever created, for any volume, ever — which means even if external-attacher were deployed as a third sidecar in deploy/controller.yaml, there would be nothing for it to watch. It would sit there, connected to the same socket as localdir-csi and external-provisioner, doing precisely nothing, forever. Deploying it would cost a container's worth of memory and a line in controller-rbac.yaml in exchange for a permanently idle process — worse than doing nothing, because it would look like this driver needed something it doesn't.

This isn't a guess about how a real driver would handle a "nothing to attach" case — it's how one actually does. csi-driver-nfs, a real, maintained CSI driver for a backend that (like this book's) never needs attaching, sets attachRequired: false, does not report PUBLISH_UNPUBLISH_VOLUME, implements ControllerPublishVolume and ControllerUnpublishVolume as bare Unimplemented stubs, and ships no csi-attacher sidecar in its own manifests at all. localdir-csi's choice here is the same shape, just slightly more thorough — this chapter gave both RPCs real logic instead of leaving them as stubs, purely to show the pattern the way Chapter 4 promised. Contrast that with csi-driver-host-path, a reference driver for storage that does model a real attach step: it leaves attachRequired unset (which defaults to true), does report PUBLISH_UNPUBLISH_VOLUME, and does deploy csi-attacher. Same spec, same RPC signatures, genuinely different deployment — because the backends are genuinely different, and each driver's manifests say so honestly.

Chapter 2's original sketch of deploy/controller.yaml as "driver + provisioner + attacher sidecars" turns out to have been a reasonable early guess that this chapter overturns, and it's worth being direct about that rather than quietly ignoring it: controller.yaml still has exactly the two containers Chapter 7 gave it. Nothing here adds a third.

Proving it, with grpcurl

Same shape as Chapter 7's proof for CreateVolume/DeleteVolume — kubectl cp the same grpcurl-linux binary into the Controller pod, kubectl exec into the localdir-csi container specifically. First, create a real volume to publish, the same way Chapter 7 did:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "attach-demo",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

{
  "volume": {
    "capacityBytes": "1048576",
    "volumeId": "attach-demo"
  }
}

Now publish it — csi-dev-control-plane is this single-node kind cluster's actual node name, the same value NodeGetInfo returns for real, via the NODE_NAME Downward API field Chapter 5 wired up:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo",
  "node_id": "csi-dev-control-plane",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:///csi/csi.sock csi.v1.Controller/ControllerPublishVolume

{}

0 OK, empty response — TestControllerPublishVolume_Succeeds, proved against a real socket. Call it again, identical request:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo",
  "node_id": "csi-dev-control-plane",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:///csi/csi.sock csi.v1.Controller/ControllerPublishVolume

{}

Still 0 OK — TestControllerPublishVolume_Idempotent. Now unpublish, deliberately omitting node_id to exercise the optional case directly:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/ControllerUnpublishVolume

{}

And once more, against a volume ID that's already unpublished:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/ControllerUnpublishVolume

{}

Both 0 OK — the same forgiving idempotency the spec text asked for, now demonstrated against a real driver process instead of just a table of test assertions. Clean up before moving on:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteVolume

Part two: closing the loop, for real

Every RPC call in this book so far has been typed by a person into grpcurl. That was never the end goal — it was a way to prove each piece works in isolation before wiring it into the machinery that's supposed to call it automatically. That machinery has been present in the cluster since Chapter 7 (external-provisioner) and Chapter 4/6 (node-driver-registrar, and kubelet itself) — it just hasn't had anything to react to yet, because nothing has created a PersistentVolumeClaim. This section creates one, and nothing about the next thousand words involves grpcurl at all.

StorageClass: telling Kubernetes which driver to ask

A PersistentVolumeClaim doesn't name a driver directly — it names a StorageClass, and the StorageClass names the driver. That indirection is what lets a cluster operator swap which actual storage backend a claim resolves to without every application's YAML needing to know or care.

Create deploy/storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: localdir-csi
provisioner: localdir.csi.example.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

provisioner is the one required field, and it has to match this driver's name exactly — the same string GetPluginInfo has returned since Chapter 3, and the same string external-provisioner already checked for on every PersistentVolumeClaim it's been watching since Chapter 7. reclaimPolicy: Delete (the default, but worth writing explicitly) means the underlying volume — and, per Chapter 7's DeleteVolume, the actual directory on disk — gets deleted automatically once its PersistentVolumeClaim is deleted, rather than left around for an operator to clean up by hand.

volumeBindingMode: Immediate (also the default) deserves an honest caveat rather than a silent pass. Immediate tells external-provisioner to call CreateVolume as soon as a claim appears, before Kubernetes has picked which node any pod using it will run on. That's fine on this book's single-node kind cluster, where every possible node is the same node — but it's a real limitation for a driver that ever runs on more than one machine, since a volume created on Node A does this driver no good for a pod the scheduler puts on Node B. The correct fix for a genuinely multi-node local-storage driver is volumeBindingMode: WaitForFirstConsumer, which delays CreateVolume until a pod actually needs the claim and a node is already chosen, combined with the driver reporting a VOLUME_ACCESSIBILITY_CONSTRAINTS plugin capability and reading CreateVolumeRequest.accessibility_requirements to create the volume in the right place. localdir-csi does neither — it's a real gap, not a hidden one, and exactly the kind of thing "retargeting CreateVolume at a real storage API," the way Chapter 2 originally framed this whole project, would need to revisit.

PersistentVolumeClaim and Pod: the demo

These two objects aren't part of the driver's own deployment the way deploy/controller.yaml or deploy/node.yaml are — they're a workload using the driver, with a lifecycle that has nothing to do with the driver's own. Keep them out of deploy/, which make deploy applies in full every time; a stray demo Pod has no business being recreated on every make deploy run. Create demo-pvc.yaml and demo-pod.yaml in the project root instead:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-claim
spec:
  storageClassName: localdir-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: demo-claim

storageClassName: localdir-csi is the only line in the claim that ties it to this driver at all — everything downstream of it happens because of that one reference. claimName: demo-claim on the Pod side is what turns "a claim exists" into "a pod actually wants to mount it" — Kubernetes won't do anything with an unclaimed PersistentVolumeClaim beyond creating the underlying volume once volumeBindingMode allows it, and with Immediate binding, that happens right away regardless of whether any pod ever references the claim at all.

Apply the claim first, on its own, to watch provisioning happen with nothing else going on:

kubectl apply -f demo-pvc.yaml
kubectl get pvc demo-claim

NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
demo-claim    Pending                                       localdir-csi   2s

Pending — external-provisioner has seen the claim (its watch loop has been running since Chapter 7) but hasn't finished acting on it yet. Check its logs:

kubectl logs -l app=localdir-csi-controller -c external-provisioner --tail=20

I0821 09:12:04.100221       1 controller.go:1449] provision "default/demo-claim" class "localdir-csi": started
I0821 09:12:04.118903       1 controller.go:1546] provision "default/demo-claim" class "localdir-csi": volume "pvc-3f9a2b7e-..." provisioned
I0821 09:12:04.118940       1 controller.go:1563] provision "default/demo-claim" class "localdir-csi": succeeded

That's CreateVolume — the exact same method this book has called by hand with grpcurl since Chapter 7 — being called automatically for the first time in this book, triggered entirely by a kubectl apply. A moment later:

kubectl get pvc demo-claim
kubectl get pv

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
demo-claim    Bound    pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310    1Gi        RWO            localdir-csi   6s

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   AGE
pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310   1Gi        RWO            Delete           Bound    default/demo-claim     localdir-csi   6s

Bound on both sides — external-provisioner's receipt, the PersistentVolume, now formally tied to the claim that asked for it. Now bring in the pod:

kubectl apply -f demo-pod.yaml
kubectl get pod demo-pod

NAME       READY   STATUS    RESTARTS   AGE
demo-pod   1/1     Running   0          9s

kubelet — the same process Chapter 4 first introduced this driver to — called NodePublishVolume on its own here, no grpcurl involved, because a running pod actually referenced the claim. Prove the mount is real, not just reported as ready:

kubectl exec demo-pod -- sh -c 'echo hello from kubernetes > /data/hello.txt'
kubectl exec demo-pod -- cat /data/hello.txt

hello from kubernetes

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310/hello.txt

hello from kubernetes

Same file, visible from both inside the container and directly on the node's own disk — the same proof-from-both-sides technique Chapter 6 used for NodePublishVolume's very first bind mount, now happening because a pod spec asked for it, not because anyone typed a mount command.

Tearing it down, and watching cleanup happen too

kubectl delete pod demo-pod
kubectl delete pvc demo-claim

Deleting the pod triggers NodeUnpublishVolume (Chapter 6) the same way creating it triggered NodePublishVolume. Deleting the claim, with reclaimPolicy: Delete in effect, triggers something new:

kubectl logs -l app=localdir-csi-controller -c external-provisioner --tail=10

I0821 09:14:51.220118       1 controller.go:1704] delete "pvc-3f9a2b7e-...": started
I0821 09:14:51.231455       1 controller.go:1712] delete "pvc-3f9a2b7e-...": volume deleted

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/

Empty. DeleteVolume, called automatically, removed the same directory grpcurl removed by hand back in Chapter 7 — and the PersistentVolume object itself is gone along with it, not left behind in a Released state, because reclaimPolicy: Delete means "delete," not "release for someone to clean up later."

What you should have now

• internal/driver/controller.go: ControllerPublishVolume and ControllerUnpublishVolume, both real, both tested, neither advertised in ControllerGetCapabilities
• A clear, spec-grounded, real-world-validated reason for that last point: attachRequired: false (set back in Chapter 4) means Kubernetes never creates a VolumeAttachment object for this driver, so external-attacher — undeployed, and correctly so — would have nothing to watch even if it were running
• deploy/storageclass.yaml: a StorageClass naming this driver, with an honestly-flagged Immediate-binding limitation for any future multi-node use
• demo-pvc.yaml and demo-pod.yaml, outside deploy/ on purpose, proving — with zero grpcurl — that a PersistentVolumeClaim alone triggers CreateVolume, that a pod referencing it triggers NodePublishVolume, and that deleting the claim triggers DeleteVolume automatically
• The full loop this book has been building toward since Chapter 1, now running end to end without a single RPC typed by hand

Every RPC this driver implements now has two forms of proof behind it: a unit test with a fake dataDir, and a real kubectl apply exercising the exact same code path through the real Kubernetes control plane. Chapter 9 turns to csi-sanity, the CSI project's own conformance test suite — a way to check this driver against the spec's own expectations directly, instead of only against the specific cases this book happened to think to write.

Chapter 9: Testing with csi-sanity

Chapter 8 closed with a promise: check this driver against the CSI spec's own expectations, not just the cases this book happened to think of. This chapter keeps that promise.

The tool is csi-sanity. It's the CSI project's own conformance suite — not written by this book, not written by anyone who has ever seen localdir-csi's source code. It only knows the spec. That's exactly what makes it worth running.

What csi-sanity actually checks

Every test this book has written so far was written by this book. That matters, and it's a real limit. A test only catches what its author thought to check. If Chapter 7 never imagined a case, Chapter 7 never tested it.

csi-sanity doesn't have that limit, or at least not the same one. It's built from the CSI spec's own rules — the same spec text this book has quoted directly, chapter after chapter, when explaining why a piece of code works the way it does. csi-sanity turns those rules into real gRPC calls, made against a real running driver, and checks the responses the same way the spec says a CO would.

Say it plainly: this is the first testing in this book that doesn't come from this book. Everything before now checked localdir-csi against its own author's assumptions. This chapter checks it against the spec directly.

A driver process, with nothing else running

csi-sanity doesn't know what Kubernetes is. It has no kubectl. It never reads a manifest. It does exactly one thing: dial a gRPC endpoint, and start making CSI calls against it.

That's a smaller ask than it sounds. It's also a smaller ask than everything Chapter 4 through Chapter 8 built. No kind cluster. No sidecars. No Deployment, no DaemonSet, no ServiceAccount. Just the localdir-csi binary, running somewhere, with a socket for csi-sanity to call.

Every real CSI driver's own test suite works exactly this way. Look at csi-driver-nfs's test/sanity/run-test.sh, or csi-driver-smb's near-identical script: both start the plugin as a bare local process, point csi-sanity at its Unix socket, and run the suite — no cluster in sight. csi-test's own hack/e2e.sh does the same thing against hostpathplugin, the project's reference driver. aws-ebs-csi-driver goes one step further and skips the separate binary entirely — it imports csi-sanity's test package straight into a native Go test and runs the driver in the same process, over an in-memory socket. Different levels of ceremony, same underlying idea: run the driver, alone, and point the conformance suite at it directly.

This book already has a way to do exactly that. It's been sitting in the Makefile since Chapter 3, unused since Chapter 4:

run:
	mkdir -p ./csi ./data
	go run ./cmd/localdir-csi

Chapter 3 used this to prove GetPluginInfo worked, months before this book ever created a kind cluster. Chapter 4 then pointed main.go's defaults back at the container paths — /csi/csi.sock and /data — because that's what a real pod needs. Running make run unmodified today tries to listen on /csi/csi.sock, an absolute path that doesn't exist, and normally shouldn't exist, on a laptop outside a container.

The fix is the same one Chapter 3 used by hand, now done through the flags Chapter 4 added instead of by editing main.go again. Add this target to the Makefile, alongside run:

sanity-run:
	mkdir -p ./csi ./data
	go run ./cmd/localdir-csi \
		-endpoint unix://$(CURDIR)/csi/csi.sock \
		-data-dir $(CURDIR)/data

Same driver. Same code, unmodified. Just pointed at a socket and a directory this book's Chapter 4 self isn't using for anything else.

One socket, not two

A real production driver usually ships as two separate deployments — one Controller, one Node — sometimes even as two separate binaries. csi-sanity expects that split by default: --csi.endpoint for Identity and Node, --csi.controllerendpoint for Controller, two flags for two different sockets.

localdir-csi doesn't have that split, and Chapter 7 already said why. The Controller Deployment and the Node DaemonSet run "this exact same binary, unmodified, main.go and all. What differs between them isn't the code; it's which container each one runs in, and which sidecar sits next to it." One binary. One socket. All three services, in every pod that runs it — Controller and Node alike.

That single-socket choice means csi-sanity only needs one flag here, not two:

csi-sanity --csi.endpoint=unix://$(pwd)/csi/csi.sock

Leave --csi.controllerendpoint unset, and csi-sanity sends Controller calls to the same endpoint as everything else — which is exactly where they're already being handled, on the exact same socket sanity-run just opened.

Installing csi-sanity

go install github.com/kubernetes-csi/csi-test/v5/cmd/csi-sanity@v5.5.0

There's no prebuilt binary to download — csi-test only ships source, so go install is the only real path. It needs Go 1.25 or newer; this project's own go.mod already asks for Go 1.26, so nothing extra to install there. v5.5.0 pins CSI spec v1.12.0 internally, one point release behind the v1.13.0 this project's own go.mod already depends on. That gap is normal, not a bug to chase down — spec point releases add fields, they don't remove the ones csi-sanity already knows how to check.

What csi-sanity would find, right now, unchanged

Before running it for real, it's worth reasoning through what it will actually do — because most of it comes down to one mechanism, repeated over and over: csi-sanity asks first, checks second.

Before testing any optional RPC, csi-sanity calls ControllerGetCapabilities or NodeGetCapabilities and checks whether this driver claims to support it. Claim it, and the matching tests run for real. Don't claim it, and csi-sanity skips those tests outright — not a failure, a deliberate no-op, logged as Skip.

localdir-csi only ever claims one capability: CREATE_DELETE_VOLUME, from ControllerGetCapabilities, wired up back in Chapter 7. NodeGetCapabilities claims nothing at all — an empty list, on purpose, since Chapter 6. Ask first, checks second: with that capability list, almost the entire suite is going to skip. GetCapacity skips. ListVolumes skips. Every snapshot RPC skips. ExpandVolume, both Controller and Node sides, skips. NodeStageVolume and NodeUnstageVolume skip. And — worth calling out directly, since Chapter 8 spent an entire section explaining why — ControllerPublishVolume and ControllerUnpublishVolume skip too, even though both are real, tested, working code. csi-sanity never looks at the code. It only looks at the capability list, and this driver's capability list never mentions PUBLISH_UNPUBLISH_VOLUME. Implementing an RPC and advertising it are two separate facts. csi-sanity only ever checks the second one.

One RPC breaks that pattern completely: ValidateVolumeCapabilities. It's not gated behind any capability flag, on either side. The spec marks it mandatory, full stop, so csi-sanity calls it no matter what ControllerGetCapabilities said. Every other optional RPC gets a graceful Skip when unsupported. ValidateVolumeCapabilities doesn't get that option.

And right now, at the end of Chapter 8, ValidateVolumeCapabilities doesn't exist. Driver embeds csi.UnimplementedControllerServer, the same way it has since Chapter 7, and that embedded type's ValidateVolumeCapabilities method does exactly one thing: return codes.Unimplemented. csi-sanity expects InvalidArgument, or a successful confirmed response, or NotFound, depending on which case it's exercising. It never expects Unimplemented. Run csi-sanity against this driver today, and this is the one place it wouldn't skip and wouldn't pass. It would fail, honestly and correctly, because the code really doesn't do what the spec says it must.

That's this chapter's real work: close that one gap, the same test-first way every other RPC in this book got built.

What a volume capability actually is

Before the code, the concept. A VolumeCapability is really just two questions, bundled into one struct.

The first question: mount, or block? A Mount capability wants a filesystem — somewhere a pod can read and write ordinary files, the way localdir-csi has worked since Chapter 4. A Block capability wants a raw block device instead, handed to the pod with no filesystem on it at all. Some databases ask for that, for the extra speed. localdir-csi never will. mountCapability(), the test helper this book has reused since Chapter 6, only ever builds the Mount kind — that's not an accident, it's this driver's whole scope.

The second question: who gets to use the volume, and how? That's the AccessMode. SINGLE_NODE_WRITER is the ordinary case — one node, read and write. MULTI_NODE_MULTI_WRITER is the less ordinary one — many nodes at once, all writing, the kind of promise a shared NFS-backed volume can make. The spec defines a few more besides, mostly narrower versions of the same idea, like read-only or single-node-only.

Put the two questions together, and a VolumeCapability boils down to one plain-language ask: "can this volume be used this way?" ValidateVolumeCapabilities is the RPC whose entire job is answering that ask — the one this chapter is about to build.

Implementing ValidateVolumeCapabilities, test-first

ValidateVolumeCapabilitiesRequest carries a volume_id and a list of volume_capabilities — both required. The response carries a Confirmed, present only when every requested capability is one this driver actually supports, plus an optional Message explaining why not, for when it isn't.

Two files again, the same two this book has been building since Chapter 3 and Chapter 7: tests go in internal/driver/controller_test.go, the method itself goes in internal/driver/controller.go. Every code block below names which one it belongs in.

Validation first, same shape as every earlier RPC in this book. Add this test to internal/driver/controller_test.go:

func TestValidateVolumeCapabilities_Validation(t *testing.T) {
	cases := []validationCase[*csi.ValidateVolumeCapabilitiesRequest]{
		{
			name: "missing volume id",
			req: &csi.ValidateVolumeCapabilitiesRequest{
				VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
			},
		},
		{
			name: "missing volume capabilities",
			req: &csi.ValidateVolumeCapabilitiesRequest{
				VolumeId: "vol-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.ValidateVolumeCapabilitiesRequest) error {
			_, err := d.ValidateVolumeCapabilities(t.Context(), req)
			return err
		},
	)
}

Same validationCase[T]/runValidation pair every validation table since Chapter 6 has used — nothing new to wire up, just another RPC whose required-field checks fit the shape they were built for.

--- FAIL: TestValidateVolumeCapabilities_Validation (0.00s)
    --- FAIL: TestValidateVolumeCapabilities_Validation/missing_volume_id (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: VolumeCapabilities:[0xc00000e5e8]})
    --- FAIL: TestValidateVolumeCapabilities_Validation/missing_volume_capabilities (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 VolumeCapabilities:[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Unimplemented — exactly the code a real csi-sanity run against the promoted stub would see too, for the same reason: nothing here validates its arguments yet, it just happens to fail with a more specific code than "missing" would. And once again, both failures land on testing_test.go:85, not on this test's own call to runValidation — the same t.Helper() mechanic Chapters 6 through 8 already established.

Minimal fix, same shallow required-field checks every other RPC in this book already uses. Add the method to internal/driver/controller.go:

func (d *Driver) ValidateVolumeCapabilities(
	ctx context.Context,
	req *csi.ValidateVolumeCapabilitiesRequest,
) (*csi.ValidateVolumeCapabilitiesResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if len(req.GetVolumeCapabilities()) == 0 {
		return nil, status.Error(codes.InvalidArgument, "volume_capabilities is required")
	}

	return &csi.ValidateVolumeCapabilitiesResponse{}, nil
}

=== RUN   TestValidateVolumeCapabilities_Validation
--- PASS: TestValidateVolumeCapabilities_Validation (0.00s)

Green. Next question, same one ControllerPublishVolume already answered in Chapter 8: what about a volume_id that's well-formed, but names nothing real? Next test, still in internal/driver/controller_test.go:

func TestValidateVolumeCapabilities_VolumeNotFound(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ValidateVolumeCapabilitiesRequest{
		VolumeId:           "does-not-exist",
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}

	_, err := d.ValidateVolumeCapabilities(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

--- FAIL: TestValidateVolumeCapabilities_VolumeNotFound (0.00s)
    controller_test.go:296: code = OK, want NotFound (req=&{VolumeId:does-not-exist VolumeCapabilities:[0xc00000e6c0]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Red — nothing checks the volume actually exists yet. The fix reaches for the exact same existence check ControllerPublishVolume already uses. Back in internal/driver/controller.go, inside ValidateVolumeCapabilities:

	path := filepath.Join(d.dataDir, req.GetVolumeId())
	if _, err := os.Stat(path); os.IsNotExist(err) {
		return nil, status.Errorf(codes.NotFound, "volume %q not found", req.GetVolumeId())
	} else if err != nil {
		return nil, status.Errorf(codes.Internal, "checking volume %q: %v", req.GetVolumeId(), err)
	}

=== RUN   TestValidateVolumeCapabilities_VolumeNotFound
--- PASS: TestValidateVolumeCapabilities_VolumeNotFound (0.00s)

Green — and worth a short pause here, not a long one. Chapter 8 already named the reason this check works at all: Controller and every Node pod share one physical disk, in this single-node kind cluster, through the same hostPath. That's still true here. It hasn't changed. It's just resurfacing, in a third method now, the same way Chapter 8 flagged it resurfacing in a second.

Now the part this chapter actually exists for: what does "supported" mean, for a capability? Start with the case that should work — a vol-1 that exists, asked about with the exact shape every other test in this book already builds through mountCapability(). Another test, in internal/driver/controller_test.go:

func TestValidateVolumeCapabilities_ConfirmsSupportedCapability(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ValidateVolumeCapabilitiesRequest{
		VolumeId:           "vol-1",
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}

	resp, err := d.ValidateVolumeCapabilities(t.Context(), req)
	if err != nil {
		t.Fatalf("ValidateVolumeCapabilities() returned an error: %v", err)
	}
	if resp.Confirmed == nil {
		t.Fatal("Confirmed is nil, want it set for a supported capability")
	}
	if len(resp.Confirmed.VolumeCapabilities) != 1 {
		t.Fatalf("Confirmed.VolumeCapabilities has %d entries, want 1", len(resp.Confirmed.VolumeCapabilities))
	}
}

--- FAIL: TestValidateVolumeCapabilities_ConfirmsSupportedCapability (0.00s)
    controller_test.go:315: Confirmed is nil, want it set for a supported capability
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.008s
FAIL

Red, correctly — the existing code returns an empty response no matter what. Here's the moment worth slowing down for. It's tempting to write the whole real implementation in one pass: echo back Confirmed when a capability is supported, and reject it with a Message when it isn't. Both behaviors, one edit. That's exactly the mistake Chapter 8 already made once, with ControllerUnpublishVolume — write two behaviors before either one has its own red test, and the second behavior never actually gets proven red at all.

So: only the minimum needed to pass this test. Back in internal/driver/controller.go:

	return &csi.ValidateVolumeCapabilitiesResponse{
		Confirmed: &csi.ValidateVolumeCapabilitiesResponse_Confirmed{
			VolumeCapabilities: req.GetVolumeCapabilities(),
		},
	}, nil

=== RUN   TestValidateVolumeCapabilities_ConfirmsSupportedCapability
--- PASS: TestValidateVolumeCapabilities_ConfirmsSupportedCapability (0.00s)

Green — but notice what that implementation actually does: it confirms everything, unconditionally. Nothing has checked whether the capability is one this driver genuinely supports. The next test exists specifically to catch that. Add it to internal/driver/controller_test.go:

This is also the first test in the book that needs a VolumeCapability with anything other than mountCapability()'s own hardcoded SINGLE_NODE_WRITER. Rather than build one by hand inline, split mountCapability() so the access mode is a parameter, in internal/driver/testing_test.go:

func mountCapability() *csi.VolumeCapability {
	return mountCapabilityWithMode(csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER)
}

func mountCapabilityWithMode(mode csi.VolumeCapability_AccessMode_Mode) *csi.VolumeCapability {
	return &csi.VolumeCapability{
		AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
		AccessMode: &csi.VolumeCapability_AccessMode{
			Mode: mode,
		},
	}
}

mountCapability() keeps every existing call site working exactly as before — it's now just a one-line wrapper around the more general helper, the same Open/Closed shape Chapter 3 and Chapter 7 both already pointed out elsewhere in this driver: the existing, narrower helper didn't need to change, only grow a sibling.

func TestValidateVolumeCapabilities_RejectsUnsupportedAccessMode(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ValidateVolumeCapabilitiesRequest{
		VolumeId: "vol-1",
		VolumeCapabilities: []*csi.VolumeCapability{
			mountCapabilityWithMode(csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER),
		},
	}

	resp, err := d.ValidateVolumeCapabilities(t.Context(), req)
	if err != nil {
		t.Fatalf("ValidateVolumeCapabilities() returned an error: %v", err)
	}
	if resp.Confirmed != nil {
		t.Fatal("Confirmed is set, want nil for an unsupported access mode")
	}
	if resp.Message == "" {
		t.Error("Message is empty, want an explanation of why the capability wasn't confirmed")
	}
}

--- FAIL: TestValidateVolumeCapabilities_RejectsUnsupportedAccessMode (0.00s)
    controller_test.go:340: Confirmed is set, want nil for an unsupported access mode
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.010s
FAIL

Real red, against the "confirm everything" version above — proof the earlier green wasn't quietly covering this case too. Now the actual supported-capability check, the first place in this entire driver that looks at which AccessMode a request names, instead of only checking that one was present at all. Last edit to internal/driver/controller.go for this chapter:

	for _, c := range req.GetVolumeCapabilities() {
		if c.GetMount() == nil {
			return &csi.ValidateVolumeCapabilitiesResponse{
				Message: "only mount volumes are supported, not block volumes",
			}, nil
		}
		if c.GetAccessMode().GetMode() != csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER {
			return &csi.ValidateVolumeCapabilitiesResponse{
				Message: "only SINGLE_NODE_WRITER is supported",
			}, nil
		}
	}

=== RUN   TestValidateVolumeCapabilities_ConfirmsSupportedCapability
--- PASS: TestValidateVolumeCapabilities_ConfirmsSupportedCapability (0.00s)
=== RUN   TestValidateVolumeCapabilities_RejectsUnsupportedAccessMode
--- PASS: TestValidateVolumeCapabilities_RejectsUnsupportedAccessMode (0.00s)

Both green. Chapter 6 flagged this moment a long way back, describing mountCapability()'s SINGLE_NODE_WRITER choice: "AccessMode starts to matter once a real StorageClass and PersistentVolumeClaim can specify one." A StorageClass and PersistentVolumeClaim have existed since Chapter 8. But nothing actually read the access mode a claim requested until just now. This is that moment, several chapters late, arriving through the one RPC the spec never lets a driver skip.

Where does "supported" actually get decided, though? Not the way ControllerGetCapabilities decides RPC support, back in Chapter 7 — that one hands back an explicit list, and csi-sanity checks it before a single test runs. VolumeCapability support has no equivalent list anywhere on Driver. It lives entirely inside the loop just written, inline, in ValidateVolumeCapabilities itself. Nothing else in this driver currently needs to ask the same question, so nothing else needs the rule duplicated. If a later chapter taught CreateVolume to reject an unsupported capability too, not just report on one, that would be the moment to pull this loop out into its own function — something both methods could call, so the rule has one home instead of two copies quietly drifting apart. Right now, one method asks the question, so one method owns the answer.

Notice, too, what's deliberately not supported here. MULTI_NODE_MULTI_WRITER gets rejected, not confirmed — even though, in this book's single-node kind cluster, every node already shares the exact same directory through the exact same hostPath. Multiple pods really could read and write it concurrently right now, and nothing would technically stop them. That's not the same as this driver promising multi-node access, though. A promise like that has to hold up against real, independent nodes — separate machines, separate disks — not just against one kind node's own implementation detail. Nothing in this driver has ever been tested against that harder case, so nothing here claims to support it. Honest scope, not a missed optimization: the same principle Chapter 7 and Chapter 8 both already leaned on when naming this driver's own local-only limits out loud.

Proving it, with grpcurl

Same two-step proof this book has used since Chapter 3: a real call, against a real running process, over the real socket. Create a volume first:

grpcurl -plaintext -unix -d '{
  "name": "sanity-demo",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' ./csi/csi.sock csi.v1.Controller/CreateVolume

{
  "volume": {
    "capacityBytes": "1048576",
    "volumeId": "sanity-demo"
  }
}

Ask whether SINGLE_NODE_WRITER mount access is valid for it:

grpcurl -plaintext -unix -d '{
  "volume_id": "sanity-demo",
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' ./csi/csi.sock csi.v1.Controller/ValidateVolumeCapabilities

{
  "confirmed": {
    "volumeCapabilities": [{"mount": {}, "accessMode": {"mode": "SINGLE_NODE_WRITER"}}]
  }
}

Confirmed. Now the same call, asking for MULTI_NODE_MULTI_WRITER instead:

grpcurl -plaintext -unix -d '{
  "volume_id": "sanity-demo",
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "MULTI_NODE_MULTI_WRITER"}}]
}' ./csi/csi.sock csi.v1.Controller/ValidateVolumeCapabilities

{
  "message": "only SINGLE_NODE_WRITER is supported"
}

No confirmed field at all, and — worth noticing directly — no error either. 0 OK, an empty-ish success response with a message explaining why. That's the spec's actual contract for this exact case, not a shortcut this driver invented: an unsupported capability is a normal, well-formed answer, not a failure.

Running csi-sanity for real

One thing to get right before starting either terminal. NodePublishVolume performs a real Linux bind mount — the exact syscall Chapter 6 needed privileged: true for, inside the kind node. Outside a pod, running as a bare process on a laptop, that same syscall still needs root. Skip this, and the Node-service half of the suite fails with mount: ... must be superuser to use mount — a permissions problem, not a driver bug. Say it plainly: the driver is fine, the process just isn't allowed to mount anything yet.

Root has to cover both terminals, not just one. The driver's Unix socket gets created by whichever user starts the driver. Start it with sudo, and the socket file ends up owned by root, with no write permission for anyone else. Connecting to a Unix socket needs write permission on that file. Run csi-sanity afterward as an ordinary user, and the connection gets refused — not because the driver rejected the call, but because the operating system never let the call reach the socket at all. The fix is symmetric: root creates the socket, so root has to be the one dialing it too.

One more wrinkle, and it's sudo's, not this driver's. sudo resets PATH by default, to a short, fixed list of system directories. go and csi-sanity almost never live in that list — they live wherever go install put them, under the calling user's own home directory. Run sudo go run ... or sudo csi-sanity ... directly, and root's shell answers command not found, even though the exact same command works fine one line earlier without sudo. env "PATH=$PATH" fixes that: it hands root's shell the calling user's own PATH, so the same go and csi-sanity root just failed to find are the first ones it finds.

Two terminals, the same pattern Chapter 3 started. First terminal, run as root, with the calling user's own PATH carried along:

sudo env "PATH=$PATH" make sanity-run

localdir-csi listening on /home/you/localdir-csi/csi/csi.sock

Second terminal, root again, for the same reason — the socket only accepts a caller with write permission on it, and that means root on both sides:

sudo env "PATH=$PATH" csi-sanity --csi.endpoint=unix://$(pwd)/csi/csi.sock

csi-sanity prints a running Ginkgo log as it works through the suite, one Describe block at a time, then a summary at the end. The shape of that summary follows directly from everything reasoned through above — worth restating plainly, since it's the whole point of this chapter:

Skipped, because ControllerGetCapabilities never claims them: GetCapacity, ListVolumes, every CreateSnapshot/ListSnapshots Describe block, ControllerExpandVolume, ControllerGetVolumeHealth and ControllerListVolumeHealth, ModifyVolume — and, still, even now, ControllerPublishVolume/ControllerUnpublishVolume, exactly as Chapter 8 predicted. Real code, real tests, real grpcurl proof behind both — none of it visible to csi-sanity, because PUBLISH_UNPUBLISH_VOLUME was never added to the capability list, on purpose, for the reasons Chapter 8 spent a whole section defending.

Skipped, because NodeGetCapabilities claims nothing at all: NodeStageVolume, NodeUnstageVolume, NodeExpandVolume, NodeGetVolumeStats, both node health RPCs.

Passing, because they're both implemented and advertised: DeleteVolume, ControllerGetCapabilities, NodeGetCapabilities, NodeGetInfo — plus every purely argument-validation It under NodePublishVolume and NodeUnpublishVolume, since checking a required field doesn't need any capability flag to be true first. Most of CreateVolume passes here too, with one honest exception, named directly below.

And passing now, for the first time, having been the one real failure this chapter set out to fix: every ValidateVolumeCapabilities case. Required-field checks, the not-found volume, the confirmed capability, the rejected one — all four, exercised by a test suite that has never seen this driver's own source code, agreeing with the driver's own unit tests written a few pages back.

Three gaps are worth naming honestly, not glossing past.

First, and different in kind from the other two — this one is a real driver gap, not a suite limitation. CreateVolume's idempotency check never looks at capacity. Call it twice for the same name, the second time asking for a different size, and the spec says that MUST fail with ALREADY_EXISTS. This driver's CreateVolume doesn't check whether a volume by that name already exists at all before creating one — it just runs os.MkdirAll, harmless the second time since the directory's already there, and returns whatever capacity this request asked for, not whatever the first one did. csi-sanity says so directly: CreateVolume [It] should fail when requesting to create a volume with already existing name and different capacity — Expected an error to have occurred. Got: nil. Chapter 7 already named this honestly, calling CreateVolume's idempotency "narrower than the full spec." This is that same gap, now confirmed by a real failing test, not just predicted.

Second: nothing in csi-sanity tests ControllerGetVolume or ControllerModifyVolume at all — not a skip, not a pass, just no test case exists for either one in this version of the suite.

Third: exact It counts and descriptions can shift a little release to release, since csi-test is still an actively maintained project — the reasoning above is what determines the shape of the result, and that shape is what's worth trusting, more than any single number.

Closing the capacity gap, for real

The first gap above isn't staying open. Chapter 6 already showed what closing a gap like this looks like, for NodePublishVolume's access-mode conflict — a metadata-only "does this exist" check can't answer "does this exist as this." CreateVolume gets the same treatment now, with a real failing csi-sanity case to prove the fix against, not just a suspicion.

Add the case that exposes it to internal/driver/controller_test.go, right after TestCreateVolume_Idempotent:

func TestCreateVolume_RejectsCapacityMismatch(t *testing.T) {
	dataDir := t.TempDir()
	d := newTestDriver(t, dataDir, nil)

	first := &csi.CreateVolumeRequest{
		Name:               "vol-1",
		CapacityRange:      &csi.CapacityRange{RequiredBytes: 1 << 20},
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}
	if _, err := d.CreateVolume(t.Context(), first); err != nil {
		t.Fatalf("first CreateVolume() returned an error: %v", err)
	}

	second := &csi.CreateVolumeRequest{
		Name:               "vol-1",
		CapacityRange:      &csi.CapacityRange{RequiredBytes: 2 << 20},
		VolumeCapabilities: []*csi.VolumeCapability{mountCapability()},
	}
	_, err := d.CreateVolume(t.Context(), second)
	requireStatusCode(t, err, codes.AlreadyExists, second)
}

go test ./internal/driver/...

Red — the exact shape csi-sanity just reported, now reproduced in a plain unit test:

=== RUN   TestCreateVolume_RejectsCapacityMismatch
    controller_test.go:125: code = OK, want AlreadyExists (req=&{Name:vol-1 CapacityRange:0xc0000a4170 VolumeCapabilities:[0xc0000da090] Parameters:map[] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil>})
--- FAIL: TestCreateVolume_RejectsCapacityMismatch (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.009s
FAIL

code = OK — a second, larger request for the same volume name just gets waved through. os.MkdirAll succeeds whether the directory already exists or not, so nothing in the current code has any way to notice a conflict — the same root cause, and the same fix, Chapter 6 already used for NodePublishVolume. Give CreateVolume its own metadata file, in internal/driver/controller.go: first the metadata type and its two helpers, matching nodePublishMeta's shape from Chapter 6 —

// volumeMetaPath is where CreateVolume records the capacity a volume was
// created with, so a later CreateVolume call for the same name can tell a
// genuine idempotent retry (same capacity) apart from a real conflict
// (different capacity).
func volumeMetaPath(dataDir, volumeID string) string {
	return filepath.Join(dataDir, ".csi-meta", "volumes", volumeID+".json")
}

// volumeMeta is the metadata CreateVolume records for each volume it
// creates.
type volumeMeta struct {
	CapacityBytes int64 `json:"capacity_bytes"`
}

func writeVolumeMeta(dataDir, volumeID string, meta volumeMeta) error {
	metaBytes, err := json.Marshal(meta)
	if err != nil {
		return err
	}
	metaPath := volumeMetaPath(dataDir, volumeID)
	if err := os.MkdirAll(filepath.Dir(metaPath), 0o750); err != nil {
		return err
	}
	return os.WriteFile(metaPath, metaBytes, 0o640)
}

func readVolumeMeta(dataDir, volumeID string) (volumeMeta, error) {
	var meta volumeMeta
	data, err := os.ReadFile(volumeMetaPath(dataDir, volumeID))
	if err != nil {
		return meta, err
	}
	err = json.Unmarshal(data, &meta)
	return meta, err
}

Add "encoding/json" to controller.go's imports if it isn't there yet. Then CreateVolume itself, replacing its old body:

func (d *Driver) CreateVolume(
	ctx context.Context,
	req *csi.CreateVolumeRequest,
) (*csi.CreateVolumeResponse, error) {
	if req.GetName() == "" {
		return nil, status.Error(codes.InvalidArgument, "name is required")
	}
	if len(req.GetVolumeCapabilities()) == 0 {
		return nil, status.Error(codes.InvalidArgument, "volume_capabilities is required")
	}

	requestedBytes := req.GetCapacityRange().GetRequiredBytes()

	if existing, err := readVolumeMeta(d.dataDir, req.GetName()); err == nil {
		if existing.CapacityBytes != requestedBytes {
			return nil, status.Errorf(codes.AlreadyExists, "volume %q already exists with a different capacity", req.GetName())
		}
		// Same name, same capacity: a genuine retry. Return the recorded
		// volume instead of re-running os.MkdirAll — MkdirAll would
		// succeed either way, silently papering over a real conflict the
		// way it did before this check existed.
		return &csi.CreateVolumeResponse{
			Volume: &csi.Volume{
				VolumeId:      req.GetName(),
				CapacityBytes: existing.CapacityBytes,
			},
		}, nil
	} else if !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "reading volume %q metadata: %v", req.GetName(), err)
	}

	path := filepath.Join(d.dataDir, req.GetName())
	if err := os.MkdirAll(path, 0o750); err != nil {
		return nil, status.Errorf(codes.Internal, "creating volume %q: %v", req.GetName(), err)
	}

	meta := volumeMeta{CapacityBytes: requestedBytes}
	if err := writeVolumeMeta(d.dataDir, req.GetName(), meta); err != nil {
		return nil, status.Errorf(codes.Internal, "recording volume %q capacity: %v", req.GetName(), err)
	}

	return &csi.CreateVolumeResponse{
		Volume: &csi.Volume{
			VolumeId:      req.GetName(),
			CapacityBytes: meta.CapacityBytes,
		},
	}, nil
}

DeleteVolume needs one more line, in the same file, cleaning up the new metadata file the same way NodeUnpublishVolume already cleans up nodePublishMeta:

	if err := os.Remove(volumeMetaPath(d.dataDir, req.GetVolumeId())); err != nil && !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "deleting volume %q metadata: %v", req.GetVolumeId(), err)
	}

That goes right after the existing os.RemoveAll(path) call, before DeleteVolume's final return.

go test ./internal/driver/...

=== RUN   TestCreateVolume_RejectsCapacityMismatch
--- PASS: TestCreateVolume_RejectsCapacityMismatch (0.00s)

Green — and TestCreateVolume_Idempotent, the test that's been there since Chapter 7, still passes too. It always retried with the same capacity, so it never exercised this gap; it just never proved this code was right. A passing idempotency test only proves what it actually retries with — worth remembering the next time a test in this book claims something "works" without saying exactly what it tried.

What you should have now

• internal/driver/controller.go: a real ValidateVolumeCapabilities, built the same one-test-at-a-time way as every other RPC in this book, including one genuine TDD misstep — two behaviors written before either had its own red test — caught by writing the second test anyway and watching it prove the gap was real
• The first code in this driver that actually inspects an AccessMode value, rather than only checking that one was present
• A deliberately narrow definition of "supported": Mount volumes, SINGLE_NODE_WRITER only — honest about what's never been tested, not just about what's technically possible on one shared kind node
• Makefile's sanity-run target: this driver, running alone, with no Kubernetes anywhere in sight — the same bare-process pattern every real CSI driver's own CI already uses to run this exact suite
• A real csi-sanity run against a real socket, whose skip/pass/fail shape now matches, case for case, everything Chapters 7 and 8 already decided and explained about this driver's own scope
• CreateVolume's own metadata file (volumeMeta, writeVolumeMeta, readVolumeMeta), closing the exact gap Chapter 7 named and this chapter's own csi-sanity run confirmed for real — the second use of the sidecar-metadata pattern Chapter 6 started

Two honest gaps remain, both already named directly above: ControllerPublishVolume/ControllerUnpublishVolume stay invisible to this suite, by choice; and ControllerGetVolume/ControllerModifyVolume stay untested by it entirely, not by choice. CreateVolume's capacity gap doesn't join them — it's closed, right above. Chapter 10 turns to snapshots — CreateSnapshot, DeleteSnapshot, and the external-snapshotter sidecar this book hasn't needed until now.

Chapter 10: Snapshots

A whiteboard sits in a conference room, covered in a diagram someone's been erasing and redrawing all week. Photocopy it right now, and the copy is fixed forever — exactly what the board looked like at 2:14pm on Tuesday. Erase half the board on Wednesday. Draw something completely different. The photocopy doesn't know. It never will. It isn't connected to the whiteboard anymore — it never was, really. It's paper, holding a pattern of light and dark that used to match a whiteboard, at one specific moment.

That's what CreateSnapshot does to a volume. A photocopy, not a link.

What a snapshot actually is, and what it's for

A PersistentVolume, once bound, just keeps existing — read from, written to, mounted into whatever pod needs it next, with no built-in notion of "before" or "after." A snapshot cuts across that. It freezes one moment of a volume's contents into a new, independent object, one that keeps existing even after the original volume changes, or gets deleted entirely.

Two real reasons that matters, both worth naming plainly. Backup: take a snapshot before a risky migration, and a bad migration has an exact undo point, not "whatever the last full backup happened to catch." Cloning: hand a snapshot to CreateVolume as a starting point (the volume_content_source field Chapter 7 already listed among CreateVolumeRequest's fields, and left untouched) and a brand-new volume starts out with another volume's exact contents, rather than empty — a fast way to spin up a database replica or a test environment seeded with real-looking data.

CSI's answer to both is CreateSnapshot and DeleteSnapshot, the two RPCs this chapter implements, plus a whole extra sidecar and two more Kubernetes CRDs to trigger them automatically — more new machinery in one chapter than Chapter 8 needed, because unlike attaching, snapshots aren't a "this driver happens not to need it" gap. localdir-csi's volumes are just directories, and directories can absolutely be copied. This is real, useful work for a toy driver to do.

Driver needs nothing new

Say it the way Chapter 7 said it about NewDriver: nothing here changes. Driver has embedded csi.UnimplementedControllerServer since Chapter 7, and ControllerServer's interface has always included CreateSnapshot, DeleteSnapshot, and ListSnapshots — Chapter 9's ValidateVolumeCapabilities work already made Driver satisfy the full interface, stubs and all. CreateSnapshot needs exactly one thing Driver already has: dataDir, the same field CreateVolume has relied on since Chapter 7. No new constructor parameter, no new embedded interface, no second red/green cycle just to catch up.

What CreateSnapshotRequest asks for, and what a Snapshot has to remember

CreateSnapshotRequest carries source_volume_id and name as its two REQUIRED fields — a snapshot always names both the volume it's copying and the name a CO wants to identify it by, the exact same name/capabilities shape CreateVolumeRequest asked for back in Chapter 7. secrets and parameters exist too, unused here, same as CreateVolumeRequest's own unused fields.

The response side is where snapshots ask for more than a volume did. A Snapshot carries snapshot_id, source_volume_id, size_bytes, ready_to_use, and creation_time — a real timestamp, not just an identifier. CreateVolume never had to remember anything like that about a volume; a directory's existence was the entire fact worth tracking. A snapshot needs history: which volume it came from, and when the copy was taken, permanently — not just for one response, but for every later CreateSnapshot call asking about the same name.

That's the same shape of problem Chapter 7 named honestly — "a bare directory on disk doesn't remember what capacity or parameters it was created with, only that it exists" — and Chapter 9's real csi-sanity run went on to confirm for real. CreateSnapshot doesn't get to leave that gap open. Answering source_volume_id and creation_time correctly on a second call requires persisting something beyond a directory's mere presence — so this chapter reaches for the same sidecar-metadata-file technique Chapter 6 introduced for NodePublishVolume, and Chapter 9 just gave CreateVolume, applying it here from day one instead of discovering the need for it the hard way.

CreateSnapshot, test-first: validation

Same rhythm as every RPC so far. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_Validation(t *testing.T) {
	cases := []validationCase[*csi.CreateSnapshotRequest]{
		{
			name: "missing source volume id",
			req: &csi.CreateSnapshotRequest{
				Name: "snap-1",
			},
		},
		{
			name: "missing name",
			req: &csi.CreateSnapshotRequest{
				SourceVolumeId: "vol-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.CreateSnapshotRequest) error {
			_, err := d.CreateSnapshot(t.Context(), req)
			return err
		},
	)
}

Same validationCase[T]/runValidation pair every validation table since Chapter 6 has used.

go test ./internal/driver/...

--- FAIL: TestCreateSnapshot_Validation (0.00s)
    --- FAIL: TestCreateSnapshot_Validation/missing_source_volume_id (0.00s)
        testing_test.go:92: code = Unimplemented, want InvalidArgument (req=&{SourceVolumeId: Name:snap-1 Secrets:map[] Parameters:map[]})
    --- FAIL: TestCreateSnapshot_Validation/missing_name (0.00s)
        testing_test.go:92: code = Unimplemented, want InvalidArgument (req=&{SourceVolumeId:vol-1 Name: Secrets:map[] Parameters:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

Unimplemented — the promoted stub talking, same as every RPC's first validation test since Chapter 6: red for the right reason, CreateSnapshot doesn't exist yet. And, as in every validation table since Chapter 6, both failures land on testing_test.go:92 — one line further down than Chapter 9's testing_test.go:85, since mountCapabilityWithMode now sits between requireStatusCode and the rest of the file — not on this test's own call to runValidation.

Add the method to internal/driver/controller.go:

func (d *Driver) CreateSnapshot(
	ctx context.Context,
	req *csi.CreateSnapshotRequest,
) (*csi.CreateSnapshotResponse, error) {
	if req.GetSourceVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "source_volume_id is required")
	}
	if req.GetName() == "" {
		return nil, status.Error(codes.InvalidArgument, "name is required")
	}

	return &csi.CreateSnapshotResponse{Snapshot: &csi.Snapshot{}}, nil
}

=== RUN   TestCreateSnapshot_Validation
--- PASS: TestCreateSnapshot_Validation (0.00s)

Green. That stub return is deliberately not nil, nil — the exact same reason Chapter 7 gave CreateVolume's own stub a real, empty *Volume instead of a missing one: CreateSnapshotResponse's snapshot field is REQUIRED, and a nil one is a broken response waiting to panic the next test that reads a field off it.

CreateSnapshot, test-first: the source volume has to exist

CreateSnapshot naming a volume that was never created — a typo, a stale reference, a volume already deleted — is a real case, the same shape ControllerPublishVolume already had to handle in Chapter 8. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_SourceVolumeNotFound(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "does-not-exist",
		Name:           "snap-1",
	}

	_, err := d.CreateSnapshot(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

--- FAIL: TestCreateSnapshot_SourceVolumeNotFound (0.00s)
    controller_test.go:376: code = OK, want NotFound (req=&{SourceVolumeId:does-not-exist Name:snap-1 Secrets:map[] Parameters:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Red — nothing checks the source volume exists yet. Same filepath.Join(d.dataDir, ...)-and-os.Stat pattern this book has reused since NodePublishVolume. Add it to CreateSnapshot, in internal/driver/controller.go, right after the validation checks:

	sourcePath := filepath.Join(d.dataDir, req.GetSourceVolumeId())
	if _, err := os.Stat(sourcePath); os.IsNotExist(err) {
		return nil, status.Errorf(codes.NotFound, "source volume %q not found", req.GetSourceVolumeId())
	} else if err != nil {
		return nil, status.Errorf(codes.Internal, "checking source volume %q: %v", req.GetSourceVolumeId(), err)
	}

=== RUN   TestCreateSnapshot_SourceVolumeNotFound
--- PASS: TestCreateSnapshot_SourceVolumeNotFound (0.00s)

Green.

CreateSnapshot, test-first: actually creating one

The real work. A valid request against a real volume should copy that volume's files somewhere new, and hand back a Snapshot describing what it copied. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_CreatesTheSnapshot(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	if err := os.WriteFile(filepath.Join(dataDir, "vol-1", "hello.txt"), []byte("hello from vol-1"), 0640); err != nil {
		t.Fatalf("writing fake volume data: %v", err)
	}
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}

	resp, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("CreateSnapshot() returned an error: %v", err)
	}

	snap := resp.GetSnapshot()
	if snap.GetSnapshotId() != "snap-1" {
		t.Errorf("SnapshotId = %q, want %q", snap.GetSnapshotId(), "snap-1")
	}
	if snap.GetSourceVolumeId() != "vol-1" {
		t.Errorf("SourceVolumeId = %q, want %q", snap.GetSourceVolumeId(), "vol-1")
	}
	if !snap.GetReadyToUse() {
		t.Error("ReadyToUse = false, want true")
	}
	if snap.GetSizeBytes() != int64(len("hello from vol-1")) {
		t.Errorf("SizeBytes = %d, want %d", snap.GetSizeBytes(), len("hello from vol-1"))
	}
	if snap.GetCreationTime() == nil {
		t.Error("CreationTime is nil, want a real timestamp")
	}

	got, err := os.ReadFile(filepath.Join(dataDir, "snapshots", "snap-1", "hello.txt"))
	if err != nil {
		t.Fatalf("reading copied snapshot data: %v", err)
	}
	if string(got) != "hello from vol-1" {
		t.Errorf("copied snapshot data = %q, want %q", got, "hello from vol-1")
	}
}

--- FAIL: TestCreateSnapshot_CreatesTheSnapshot (0.00s)
    controller_test.go:399: SnapshotId = "", want "snap-1"
    controller_test.go:402: SourceVolumeId = "", want "vol-1"
    controller_test.go:405: ReadyToUse = false, want true
    controller_test.go:408: SizeBytes = 0, want 16
    controller_test.go:411: CreationTime is nil, want a real timestamp
    controller_test.go:416: reading copied snapshot data: open .../snapshots/snap-1/hello.txt: no such file or directory
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Five separate assertions failing from one missing feature — worth reading as one red, not five. Nothing about the stub touches the filesystem or the response at all yet.

Fixing this needs three things: a place on disk for a snapshot's copied files, a place on disk for a snapshot's metadata, and the copy itself. Add these three helpers below CreateSnapshot in internal/driver/controller.go:

// snapshotDataPath is where a snapshot's copied files live — a full copy of
// the source volume's directory, at the time CreateSnapshot ran.
func snapshotDataPath(dataDir, snapshotID string) string {
	return filepath.Join(dataDir, "snapshots", snapshotID)
}

// snapshotMetaPath is where a snapshot's metadata lives — deliberately
// outside snapshotDataPath, the same reasoning behind every sidecar
// metadata file this book uses: metadata must never end up inside a
// directory that later gets bind-mounted or copied again as volume data.
func snapshotMetaPath(dataDir, snapshotID string) string {
	return filepath.Join(dataDir, ".csi-meta", "snapshots", snapshotID+".json")
}

// dirSize adds up the size of every regular file under path — the closest
// thing this toy driver has to asking a real backend how large a snapshot
// turned out to be.
func dirSize(path string) (int64, error) {
	var total int64
	err := filepath.WalkDir(path, func(_ string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		if d.IsDir() {
			return nil
		}
		info, err := d.Info()
		if err != nil {
			return err
		}
		total += info.Size()
		return nil
	})
	return total, err
}

dirSize walks the whole snapshot directory, not just its top level. filepath.WalkDir visits every entry under path — files and directories both — going arbitrarily deep, however many subdirectories the source volume happens to have. A volume with three levels of nested folders gets all three levels walked; dirSize doesn't know or care how deep it goes, and doesn't need to be told.

The callback answers one question per entry: is this a file? Directories return early — d.IsDir() — because a directory has no size of its own worth counting; only the files inside it do. Every regular file's size gets added to total, via d.Info().Size(). Skip a directory, add up a file: that's the whole function, run once per entry in the tree.

One more detail worth naming: the first error WalkDir hits — a permission problem, a broken symlink, anything — gets returned straight up and stops the walk immediately. dirSize never adds up a partial answer. A snapshot's reported size is either complete and correct, or CreateSnapshot fails and reports nothing at all — see the Internal error a few lines below, once this helper is wired in.

snapshotMeta, alongside them, is the piece actually worth pausing on: the data structure CreateSnapshot writes now, and reads back later. Add this too:

// snapshotMeta is the metadata CreateSnapshot records for each snapshot it
// creates, so a later CreateSnapshot call for the same name can tell a
// genuine idempotent retry (same source volume) apart from a real conflict
// (different source volume).
type snapshotMeta struct {
	SourceVolumeID string    `json:"source_volume_id"`
	CreationTime   time.Time `json:"creation_time"`
	SizeBytes      int64     `json:"size_bytes"`
}

func writeSnapshotMeta(dataDir, snapshotID string, meta snapshotMeta) error {
	metaBytes, err := json.Marshal(meta)
	if err != nil {
		return err
	}
	metaPath := snapshotMetaPath(dataDir, snapshotID)
	if err := os.MkdirAll(filepath.Dir(metaPath), 0o750); err != nil {
		return err
	}
	return os.WriteFile(metaPath, metaBytes, 0o640)
}

func readSnapshotMeta(dataDir, snapshotID string) (snapshotMeta, error) {
	var meta snapshotMeta
	data, err := os.ReadFile(snapshotMetaPath(dataDir, snapshotID))
	if err != nil {
		return meta, err
	}
	err = json.Unmarshal(data, &meta)
	return meta, err
}

Three fields, three jobs. SourceVolumeID is what makes idempotency and conflict-detection possible: a retry of CreateSnapshot for the same name only counts as the same snapshot if the source volume matches too — exactly the check CreateSnapshot makes a few lines below, once this struct exists to check against. CreationTime is captured once, the moment the snapshot is made, and never touched again; read it back a year later and it still says when the snapshot was actually created, not when someone last happened to ask. SizeBytes is dirSize's answer, cached at creation time rather than recomputed on every read — walking a directory tree on every question a caller might ask would get expensive once the driver is holding many snapshots, so CreateSnapshot pays that cost exactly once and writes the answer down.

writeSnapshotMeta and readSnapshotMeta are the same shape you've already seen: marshal snapshotMeta to JSON, write it under .csi-meta/, and on the way back in, read the bytes and unmarshal them into the same struct. Nothing new in the mechanics — this is Chapter 7's capacity-metadata pattern, reused for a second kind of metadata.

Add "encoding/json", "io/fs", and "time" to controller.go's imports.

Now the copy itself, replacing CreateSnapshot's old stub return, in internal/driver/controller.go:

	destPath := snapshotDataPath(d.dataDir, req.GetName())
	if err := os.CopyFS(destPath, os.DirFS(sourcePath)); err != nil {
		return nil, status.Errorf(codes.Internal, "copying volume %q into snapshot %q: %v", req.GetSourceVolumeId(), req.GetName(), err)
	}

	size, err := dirSize(destPath)
	if err != nil {
		return nil, status.Errorf(codes.Internal, "measuring snapshot %q: %v", req.GetName(), err)
	}

	meta := snapshotMeta{
		SourceVolumeID: req.GetSourceVolumeId(),
		CreationTime:   time.Now().UTC(),
		SizeBytes:      size,
	}
	if err := writeSnapshotMeta(d.dataDir, req.GetName(), meta); err != nil {
		return nil, status.Errorf(codes.Internal, "recording snapshot %q metadata: %v", req.GetName(), err)
	}

	return &csi.CreateSnapshotResponse{
		Snapshot: &csi.Snapshot{
			SnapshotId:     req.GetName(),
			SourceVolumeId: meta.SourceVolumeID,
			SizeBytes:      meta.SizeBytes,
			CreationTime:   timestamppb.New(meta.CreationTime),
			ReadyToUse:     true,
		},
	}, nil

Add "google.golang.org/protobuf/types/known/timestamppb" to controller.go's imports — the same package the real generated csi package already depends on for creation_time, since the spec defines that field as a google.protobuf.Timestamp, not a plain string or integer. timestamppb.New(t) builds one from an ordinary time.Time; .AsTime() is its inverse, for whenever this driver needs to read a Timestamp back as a time.Time (it doesn't, yet — nothing in this chapter converts one back). The test above checks GetCreationTime() == nil rather than .AsTime().IsZero() on purpose: a real *timestamppb.Timestamp's AsTime() converts a nil receiver to the Unix epoch, not Go's zero time.Time{} — .IsZero() on that result is false, not true, so it can't actually tell "never set" apart from "a snapshot genuinely created at 1970-01-01." Checking the pointer for nil is the one that's actually reliable.

os.CopyFS — new to this book, standard library since Go 1.23 — does the entire recursive copy in one call: os.CopyFS(dir, fsys) copies every file fsys contains into dir, creating dir itself if it doesn't already exist. os.DirFS(sourcePath) turns an ordinary directory into the fs.FS value CopyFS wants. Two standard-library calls, no hand-rolled recursive walk needed — the same "no exotic machinery" spirit this driver has held to since Chapter 1, now applying to a slightly less trivial operation than MkdirAll.

go test ./internal/driver/...

=== RUN   TestCreateSnapshot_CreatesTheSnapshot
--- PASS: TestCreateSnapshot_CreatesTheSnapshot (0.00s)

Green. ReadyToUse: true unconditionally is worth a direct word, since the field name implies there could be a false case: localdir-csi's "copy" is a single synchronous os.CopyFS call, done by the time CreateSnapshot returns — there's no window where the snapshot exists but isn't finished yet. A real backend snapshotting a multi-terabyte volume asynchronously would legitimately return ready_to_use: false at first, with ListSnapshots or a later poll eventually reporting true once the backend finishes. This driver never has that problem, so it never needs that field's honest range — another small, real consequence of "local directories on the node," the same simplification Chapter 7 already named as this driver's defining one.

CreateSnapshot, test-first: idempotency, and a conflict caught from day one

Same two-part shape the spec asked of CreateVolume back in Chapter 7: OK for a genuine retry, a real error for a genuine conflict. Add both cases at once to internal/driver/controller_test.go:

func TestCreateSnapshot_Idempotent(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	if err := os.WriteFile(filepath.Join(dataDir, "vol-1", "hello.txt"), []byte("hello"), 0o640); err != nil {
		t.Fatalf("writing fake volume data: %v", err)
	}
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}

	first, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("first CreateSnapshot() returned an error: %v", err)
	}

	second, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("second CreateSnapshot() for the same name returned an error: %v", err)
	}
	if second.GetSnapshot().GetSnapshotId() != first.GetSnapshot().GetSnapshotId() {
		t.Errorf("SnapshotId = %q, want %q", second.GetSnapshot().GetSnapshotId(), first.GetSnapshot().GetSnapshotId())
	}
	if second.GetSnapshot().GetSourceVolumeId() != "vol-1" {
		t.Errorf("SourceVolumeId = %q, want %q", second.GetSnapshot().GetSourceVolumeId(), "vol-1")
	}
}

func TestCreateSnapshot_RejectsSourceVolumeMismatch(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	makeVolumeDir(t, dataDir, "vol-2")
	d := newTestDriver(t, dataDir, nil)

	first := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}
	if _, err := d.CreateSnapshot(t.Context(), first); err != nil {
		t.Fatalf("first CreateSnapshot() returned an error: %v", err)
	}

	second := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-2",
		Name:           "snap-1",
	}
	_, err := d.CreateSnapshot(t.Context(), second)
	requireStatusCode(t, err, codes.AlreadyExists, second)
}

=== RUN   TestCreateSnapshot_Idempotent
    controller_test.go:443: second CreateSnapshot() for the same name returned an error: rpc error: code = Internal desc = copying volume "vol-1" into snapshot "snap-1": open /tmp/TestCreateSnapshot_Idempotent2485715020/001/snapshots/snap-1/hello.txt: file exists
--- FAIL: TestCreateSnapshot_Idempotent (0.00s)
=== RUN   TestCreateSnapshot_RejectsSourceVolumeMismatch
    controller_test.go:472: code = OK, want AlreadyExists (req=&{SourceVolumeId:vol-2 Name:snap-1 Secrets:map[] Parameters:map[]})
--- FAIL: TestCreateSnapshot_RejectsSourceVolumeMismatch (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

Both fail, but not for the same reason. The conflict case fails the way you'd expect — no check exists yet, so a mismatched source volume sails through as OK. The idempotency case fails for a sharper reason, worth stating plainly: os.CopyFS is not idempotent-safe. Standard library, since Go 1.23, and it refuses to overwrite a file that already exists at the destination — call it twice with the same source and destination and the second call fails with exactly the error above, file exists. A source volume with real data in it makes that collision certain on any genuine retry. TestCreateSnapshot_Idempotent seeding an actual file, instead of leaving the volume empty, is what surfaces this — an empty volume gives os.CopyFS nothing to collide with on the second call, so the test would pass by accident and hide the bug. That's the exact same lesson Chapter 9's real csi-sanity run taught about CreateVolume's own idempotency gap: a test only proves what it actually exercises.

Both gaps need the same fix: check the snapshot's own metadata before touching the filesystem at all, and short-circuit on a genuine retry instead of ever calling os.CopyFS a second time. Add the check to CreateSnapshot, in internal/driver/controller.go, right after the source-volume existence check and before the copy:

	if existing, err := readSnapshotMeta(d.dataDir, req.GetName()); err == nil {
		if existing.SourceVolumeID != req.GetSourceVolumeId() {
			return nil, status.Errorf(codes.AlreadyExists, "snapshot %q already exists for a different source volume", req.GetName())
		}
		// Same name, same source volume: a genuine retry. Return the
		// snapshot already on disk instead of falling through to
		// os.CopyFS below — it refuses to overwrite files that already
		// exist, so re-copying here would fail on every idempotent retry.
		return &csi.CreateSnapshotResponse{
			Snapshot: &csi.Snapshot{
				SnapshotId:     req.GetName(),
				SourceVolumeId: existing.SourceVolumeID,
				SizeBytes:      existing.SizeBytes,
				CreationTime:   timestamppb.New(existing.CreationTime),
				ReadyToUse:     true,
			},
		}, nil
	} else if !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "reading snapshot %q metadata: %v", req.GetName(), err)
	}

=== RUN   TestCreateSnapshot_Idempotent
--- PASS: TestCreateSnapshot_Idempotent (0.00s)
=== RUN   TestCreateSnapshot_RejectsSourceVolumeMismatch
--- PASS: TestCreateSnapshot_RejectsSourceVolumeMismatch (0.00s)

Both green — for real reasons this time, not by accident of an empty test volume. Read the metadata once, decide from it: a matching source volume is a retry, answered from what's already recorded; a mismatched one is a real conflict, rejected; anything else falls through to do the actual copy, exactly once, ever, per snapshot name.

DeleteSnapshot, test-first

Same three-cycle shape as DeleteVolume in Chapter 7. Add to internal/driver/controller_test.go:

func TestDeleteSnapshot_Validation(t *testing.T) {
	d := newTestDriverInTempDir(t)

	_, err := d.DeleteSnapshot(t.Context(), &csi.DeleteSnapshotRequest{})
	requireStatusCode(t, err, codes.InvalidArgument, &csi.DeleteSnapshotRequest{})
}

--- FAIL: TestDeleteSnapshot_Validation (0.00s)
    controller_test.go:479: code = Unimplemented, want InvalidArgument (req=&{SnapshotId: Secrets:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

func (d *Driver) DeleteSnapshot(
	ctx context.Context,
	req *csi.DeleteSnapshotRequest,
) (*csi.DeleteSnapshotResponse, error) {
	if req.GetSnapshotId() == "" {
		return nil, status.Error(codes.InvalidArgument, "snapshot_id is required")
	}

	return &csi.DeleteSnapshotResponse{}, nil
}

=== RUN   TestDeleteSnapshot_Validation
--- PASS: TestDeleteSnapshot_Validation (0.00s)

Green. Now the removal, and the idempotency case, both at once:

func TestDeleteSnapshot_RemovesTheSnapshot(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	createReq := &csi.CreateSnapshotRequest{SourceVolumeId: "vol-1", Name: "snap-1"}
	if _, err := d.CreateSnapshot(t.Context(), createReq); err != nil {
		t.Fatalf("CreateSnapshot() returned an error: %v", err)
	}

	deleteReq := &csi.DeleteSnapshotRequest{SnapshotId: "snap-1"}
	if _, err := d.DeleteSnapshot(t.Context(), deleteReq); err != nil {
		t.Fatalf("DeleteSnapshot() returned an error: %v", err)
	}

	if _, err := os.Stat(filepath.Join(dataDir, "snapshots", "snap-1")); !os.IsNotExist(err) {
		t.Errorf("snapshot data directory still exists after DeleteSnapshot")
	}
	if _, err := os.Stat(filepath.Join(dataDir, ".csi-meta", "snapshots", "snap-1.json")); !os.IsNotExist(err) {
		t.Errorf("snapshot metadata still exists after DeleteSnapshot")
	}
}

func TestDeleteSnapshot_Idempotent(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.DeleteSnapshotRequest{SnapshotId: "does-not-exist"}

	if _, err := d.DeleteSnapshot(t.Context(), req); err != nil {
		t.Fatalf("DeleteSnapshot() for an already-gone snapshot returned an error: %v", err)
	}
}

--- FAIL: TestDeleteSnapshot_RemovesTheSnapshot (0.00s)
    controller_test.go:498: snapshot data directory still exists after DeleteSnapshot
    controller_test.go:501: snapshot metadata still exists after DeleteSnapshot
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.011s
FAIL

TestDeleteSnapshot_Idempotent passes immediately — os.RemoveAll's already-gone-is-fine behavior, the same free property DeleteVolume had in Chapter 7. The removal test doesn't. Fix it in internal/driver/controller.go, replacing DeleteSnapshot's old stub return:

	if err := os.RemoveAll(snapshotDataPath(d.dataDir, req.GetSnapshotId())); err != nil {
		return nil, status.Errorf(codes.Internal, "deleting snapshot %q: %v", req.GetSnapshotId(), err)
	}
	if err := os.Remove(snapshotMetaPath(d.dataDir, req.GetSnapshotId())); err != nil && !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "deleting snapshot %q metadata: %v", req.GetSnapshotId(), err)
	}

	return &csi.DeleteSnapshotResponse{}, nil

=== RUN   TestDeleteSnapshot_RemovesTheSnapshot
--- PASS: TestDeleteSnapshot_RemovesTheSnapshot (0.00s)
=== RUN   TestDeleteSnapshot_Idempotent
--- PASS: TestDeleteSnapshot_Idempotent (0.00s)

Both green.

ControllerGetCapabilities keeps the promise Chapter 7 made

Chapter 7 built ControllerGetCapabilities around a slice literal on purpose, "specifically so that a later chapter adding LIST_VOLUMES or GET_CAPACITY support means appending an entry, not restructuring this method." This is that later chapter, just with a different capability than the two named at the time. Add this test to internal/driver/controller_test.go:

func TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot(t *testing.T) {
	d := newTestDriver(t, dataDir, nil)

	resp, err := d.ControllerGetCapabilities(t.Context(), &csi.ControllerGetCapabilitiesRequest{})
	if err != nil {
		t.Fatalf("ControllerGetCapabilities returned an error: %v", err)
	}

	var found bool
	for _, c := range resp.Capabilities {
		if rpc := c.GetRpc(); rpc != nil && rpc.Type == csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT {
			found = true
		}
	}
	if !found {
		t.Error("CREATE_DELETE_SNAPSHOT not found among advertised capabilities")
	}
}

--- FAIL: TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot (0.00s)
    controller_test.go:530: CREATE_DELETE_SNAPSHOT not found among advertised capabilities
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

Append the entry, in internal/driver/controller.go:

			{
				Type: &csi.ControllerServiceCapability_Rpc{
					Rpc: &csi.ControllerServiceCapability_RPC{
						Type: csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,
					},
				},
			},

That goes inside ControllerGetCapabilities's existing Capabilities: []*csi.ControllerServiceCapability{...} slice, right after the CREATE_DELETE_VOLUME entry Chapter 7 wrote. No restructuring, exactly as promised — one entry, appended.

go test ./internal/driver/...

--- FAIL: TestControllerGetCapabilities (0.00s)
    controller_test.go:21: got 2 capabilities, want 1
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

The promise held for the implementation — one entry, appended, no restructuring. It didn't hold for the original test. TestControllerGetCapabilities, written back in Chapter 7, hardcoded len(resp.Capabilities) != 1, a check that was never going to survive a second capability being added, appended or not. Worth being honest about that rather than quietly patching around it: Open/Closed applied to the method's own logic, not to every assertion a much earlier test happened to make about its output. Fix the old test itself, in internal/driver/controller_test.go:

	if len(resp.Capabilities) == 0 {
		t.Fatal("got 0 capabilities, want at least 1")
	}

That replaces the old != 1 check. The rest of TestControllerGetCapabilities — asserting Capabilities[0] is specifically CREATE_DELETE_VOLUME — stays exactly as Chapter 7 wrote it, since ControllerGetCapabilities still appends new entries after the existing ones, never reorders them.

=== RUN   TestControllerGetCapabilities
--- PASS: TestControllerGetCapabilities (0.00s)
=== RUN   TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot
--- PASS: TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot (0.00s)

Both green, and the full suite:

go test ./internal/driver/... -v

ok  	github.com/yourname/localdir-csi/internal/driver	0.015s

What this chapter doesn't implement, on purpose

Two gaps, named directly, not glossed past.

ListSnapshots stays a bare Unimplemented stub, the same as ControllerPublishVolume was before Chapter 8 gave it a real body — except this time, unlike Chapter 8, nothing in this chapter gives it one. ControllerGetCapabilities never advertises LIST_SNAPSHOTS alongside CREATE_DELETE_SNAPSHOT, so any conformant CO, csi-sanity included, treats it exactly the way Chapter 9 explained for every other unadvertised RPC: a silent, deliberate skip, not a failure. A CO can still create and delete snapshots by name without it; ListSnapshots only matters for discovering snapshots a CO doesn't already know the IDs of, a case this book's own demo never needs.

CreateVolume still doesn't read volume_content_source — the field that would let a new volume start out as a copy of an existing snapshot, the "cloning" half of this chapter's opening list of reasons snapshots matter. CreateSnapshot and DeleteSnapshot cover the "backup" half completely: a snapshot can be taken, inspected on disk, and removed. Restoring from one — turning a snapshot back into a usable volume — is real, additional work CreateVolume doesn't do, and this book's outline doesn't schedule a return trip to it. Worth knowing exactly where the line sits, rather than assuming "snapshots" as a chapter title means the whole feature is done.

deploy/: the first cluster-level component this book has needed

Every sidecar so far — node-driver-registrar in Chapter 4, external-provisioner in Chapter 7 — ships as a container inside one of this driver's own pods, in deploy/node.yaml or deploy/controller.yaml. Snapshots need one of those too, but they also need something genuinely new: a piece of cluster machinery that has nothing to do with localdir-csi specifically, installed once per cluster, shared by every CSI driver that ever wants to support snapshots.

That's the snapshot-controller — not a sidecar, and not part of deploy/ at all. It watches VolumeSnapshot objects cluster-wide (the CO-facing, namespaced request — "I want a snapshot of this claim," the direct analog of a PersistentVolumeClaim) and, for each one, creates a matching VolumeSnapshotContent object (the cluster-scoped record of an actual snapshot, the direct analog of a PersistentVolume). It never talks to any driver's socket directly. That's csi-snapshotter's job — the actual sidecar, watching VolumeSnapshotContent objects and turning them into the CreateSnapshot/DeleteSnapshot gRPC calls this chapter just built, the same relationship external-provisioner has to PersistentVolumes and CreateVolume.

Three CRDs come with this split: VolumeSnapshotClass (the StorageClass equivalent — names which driver handles a class of snapshot), VolumeSnapshot, and VolumeSnapshotContent. None of them exist in a stock Kubernetes cluster; they're part of what installing the snapshot-controller sets up.

Install the CRDs and the snapshot-controller once, cluster-wide — not part of make deploy, and not repeated for every driver a cluster ever adds:

kubectl kustomize "https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v8.2.0" | kubectl create -f -
kubectl -n kube-system kustomize "https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v8.2.0" | kubectl create -f -

kubectl get pods -n kube-system -l app.kubernetes.io/name=snapshot-controller

NAME                                  READY   STATUS    RESTARTS   AGE
snapshot-controller-7d8f6c9b5-k2n4p   1/1     Running   0          11s

That pod will sit there watching VolumeSnapshot objects regardless of which drivers this cluster ever adds — genuinely shared infrastructure, the first thing this book has deployed that isn't localdir-csi's own.

deploy/controller.yaml and deploy/controller-rbac.yaml: the third sidecar

csi-snapshotter, unlike snapshot-controller, is per-driver — it needs this driver's own socket, so it belongs in deploy/controller.yaml alongside localdir-csi and external-provisioner. Add a third container:

        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi

That block goes inside deploy/controller.yaml's existing containers: list, after the external-provisioner entry Chapter 7 added — same socket-dir emptyDir, same --csi-address flag shape, no new volume needed. Three containers now sharing one socket: the driver itself, and two sidecars each watching a different piece of cluster state and translating it into calls against that same socket.

RBAC needs a real extension too, not a cosmetic one. Chapter 7 trimmed external-provisioner's rules and said directly: "snapshot-related rules (volumesnapshots, volumesnapshotcontents) are absent because localdir-csi doesn't implement CreateSnapshot." That's no longer true. Add a second ClusterRole and ClusterRoleBinding to deploy/controller-rbac.yaml, trimmed from external-snapshotter's own published reference RBAC the same honest way Chapter 7 trimmed external-provisioner's:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: localdir-csi-snapshotter
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents/status"]
    verbs: ["update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: localdir-csi-snapshotter
subjects:
  - kind: ServiceAccount
    name: localdir-csi-controller
    namespace: default
roleRef:
  kind: ClusterRole
  name: localdir-csi-snapshotter
  apiGroup: rbac.authorization.k8s.io

No new ServiceAccount — the binding's subjects names localdir-csi-controller, the exact same one Chapter 7 already created. serviceAccountName in deploy/controller.yaml's pod spec only takes one value, and every container in a pod runs under that same identity regardless of which sidecar happens to need which specific permissions. One ServiceAccount, now bound to two ClusterRoles instead of one — external-provisioner's from Chapter 7, and csi-snapshotter's, both from deploy/controller-rbac.yaml — is exactly how a pod with three containers and two different sidecars' worth of Kubernetes API needs actually works.

Two categories missing from csi-snapshotter's own reference RBAC here, gone on purpose, the same way Chapter 7 explained its own omissions rather than leaving them silent. Volume group snapshot rules (volumegroupsnapshotclasses, volumegroupsnapshotcontents) — csi-snapshotter's published reference RBAC includes them, but localdir-csi doesn't implement CREATE_DELETE_VOLUME_GROUP_SNAPSHOT, a capability this book has never mentioned and doesn't start now. Leader-election rules on leases — same reasoning Chapter 7 already gave for skipping them on external-provisioner: a lock only earns its keep with more than one replica contending for it, and deploy/controller.yaml has exactly one.

deploy/volumesnapshotclass.yaml

The StorageClass equivalent for snapshots — the object that names which driver a VolumeSnapshot actually resolves to:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: localdir-csi-snapshot
driver: localdir.csi.example.com
deletionPolicy: Delete

driver has to match this driver's own name exactly, the same provisioner field rule Chapter 8's StorageClass already followed. deletionPolicy: Delete mirrors StorageClass's own reclaimPolicy: the underlying VolumeSnapshotContent — and, per this chapter's own DeleteSnapshot, the actual copied directory on disk — gets deleted automatically once its VolumeSnapshot is deleted.

Deploy everything:

make deploy
kubectl apply -f deploy/volumesnapshotclass.yaml
kubectl get pods -l app=localdir-csi-controller

NAME                                        READY   STATUS    RESTARTS   AGE
localdir-csi-controller-6f7d8b9c4d-x9m3q    3/3     Running   0          12s

3/3 — the same signal Chapter 4 and Chapter 7 both already explained, now with a third container in it.

Proving it, with grpcurl

Same kubectl cp/kubectl exec pattern as Chapters 7 and 8. First, a real volume, with real data in it — grpcurl alone only ever proved volumes exist, so this is the first proof in this book that reaches past the socket onto the node's own disk to seed content directly, before calling anything:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "snap-source",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

docker exec csi-dev-control-plane sh -c \
  'echo "hello from a real volume" > /var/lib/localdir-csi/data/snap-source/hello.txt'

Now snapshot it:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

Check the copy landed on the node's own disk, independent of anything the driver reported about itself:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/snap-demo/hello.txt

hello from a real volume

Real data, really copied. Call CreateSnapshot again, identical request:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

ERROR:
  Code: Internal
  Message: copying volume "snap-source" into snapshot "/data/snapshots/snap-demo": open /data/snapshots/snap-demo/hello.txt: file exists
command terminated with exit code 77

Not 0 OK — a real failure, against a real cluster, on the exact retry this section calls "the same request." This is the os.CopyFS-refuses-to-overwrite bug from the TDD walkthrough above, caught live: the driver running in this cluster predated the early-return fix, so a genuine retry fell through to a second copy and collided with the file the first call had already written. The fix is the one already added to internal/driver/controller.go above. make deploy (Chapter 2) rebuilds the image, reloads it into the cluster, and restarts both the Controller Deployment and the Node DaemonSet on your behalf, so the already-running pod picks up the new build without any extra command here:

make deploy

Call CreateSnapshot again, same request, against the redeployed driver:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

0 OK — and worth reading closely: creationTime is the exact same timestamp as the very first CreateSnapshot call above, down to the nanosecond. The driver's data directory is a hostPath, so it survived the redeploy — snap-demo's metadata was still sitting on disk from before the fix. This call landed on the new early-return branch, matched that recorded metadata, and returned it straight back without ever touching os.CopyFS. Confirm the file itself was never touched again either:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/snap-demo/hello.txt

hello from a real volume

Same content, same one real copy. Call CreateSnapshot a third time, same request, to prove the first retry wasn't a fluke:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

Identical again. TestCreateSnapshot_Idempotent, proved for real this time, against the exact bug that first broke it. Now delete it:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "snapshot_id": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteSnapshot

{}

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/snapshots/

Empty — gone, the same proof-from-both-sides technique every earlier chapter's grpcurl section has used.

Part two: a VolumeSnapshot object, watched and acted on automatically

Same closing move as Chapter 8's PersistentVolumeClaim section: prove the whole loop runs without a single RPC typed by hand. Reuse demo-pvc.yaml and demo-pod.yaml from Chapter 8 to get a real, mounted, localdir-csi-backed volume with a pod actually writing to it:

kubectl apply -f demo-pvc.yaml
kubectl apply -f demo-pod.yaml
kubectl exec demo-pod -- sh -c 'echo snapshot me > /data/notes.txt'

Create a VolumeSnapshot, naming the claim it's a snapshot of, and the VolumeSnapshotClass from earlier:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: demo-snapshot
spec:
  volumeSnapshotClassName: localdir-csi-snapshot
  source:
    persistentVolumeClaimName: demo-claim

Save that as demo-volumesnapshot.yaml, outside deploy/, the same place Chapter 8 kept demo-pvc.yaml — a workload object, not part of this driver's own deployment.

kubectl apply -f demo-volumesnapshot.yaml
kubectl get volumesnapshot demo-snapshot

NAME            READYTOUSE   SOURCEPVC     RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                       AGE
demo-snapshot   false        demo-claim                  localdir-csi-snapshot                                                          2s

READYTOUSE false — a moment worth watching, not skipping past. Check what's actually happening:

kubectl logs -n kube-system -l app.kubernetes.io/name=snapshot-controller --tail=10

I0821 10:22:01.334221       1 snapshot_controller.go:255] createSnapshotContent: Creating content for snapshot demo-snapshot through the plugin ...
I0821 10:22:01.335590       1 snapshot_controller.go:271] volumeSnapshotContent "snapcontent-..." created

That's snapshot-controller — cluster-level, watching VolumeSnapshot directly — creating the matching VolumeSnapshotContent object. Nothing has called CreateSnapshot yet; that's the next step, and it belongs to a different component entirely:

kubectl logs -l app=localdir-csi-controller -c csi-snapshotter --tail=10

I0821 10:22:01.410332       1 snapshotter.go:126] CreateSnapshot for content snapcontent-3a9f21e0-...
I0821 10:22:01.410451       1 controller.go:759] createSnapshotWrapper: Creating snapshot for content ...
I0821 10:22:01.421187       1 controller.go:801] createSnapshotWrapper: create snapshot ... succeeded

csi-snapshotter — per-driver, watching VolumeSnapshotContent — is what actually called CreateSnapshot, the exact method this chapter built, over the exact same socket external-provisioner already shares. A moment later:

kubectl get volumesnapshot demo-snapshot

NAME            READYTOUSE   SOURCEPVC     RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                       AGE
demo-snapshot   true         demo-claim    1Gi           localdir-csi-snapshot   snapcontent-3a9f21e0-...                              6s

READYTOUSE true — this driver's own ReadyToUse: true, set the instant CreateSnapshot returned, now visible all the way up through VolumeSnapshotContent to the VolumeSnapshot a person actually looks at. Confirm the data itself, directly on the node's disk, the same way every earlier proof in this book has:

CONTENT=$(kubectl get volumesnapshot demo-snapshot -o jsonpath='{.status.boundVolumeSnapshotContentName}')
SNAPID=$(kubectl get volumesnapshotcontent "$CONTENT" -o jsonpath='{.status.snapshotHandle}')
docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/"$SNAPID"/notes.txt

snapshot me

The exact string a pod wrote through a real mount, now sitting inside a snapshot directory nothing but a kubectl apply ever asked for by name.

Tear it down:

kubectl delete volumesnapshot demo-snapshot
docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/snapshots/

Empty — DeleteSnapshot, called automatically, the same cleanup Chapter 8 already proved for DeleteVolume when a claim gets deleted.

What you should have now

• internal/driver/controller.go: real CreateSnapshot and DeleteSnapshot, both tested, both idempotent — the conflict check came from designing the metadata in from day one, but the idempotency bug itself was found the same way Chapter 9 found CreateVolume's capacity gap: a test that passed by accident (an empty test volume hid it), caught for real once a real cluster run copied actual data
• A working sidecar-metadata-file pattern (snapshotMeta, writeSnapshotMeta, readSnapshotMeta) that answers "what was this called with, last time" for data a bare directory can't remember on its own
• ControllerGetCapabilities advertising CREATE_DELETE_SNAPSHOT, proving out the exact Open/Closed extension point Chapter 7 designed in — and one honest fix to a Chapter 7 test that wasn't quite as future-proof as its own implementation
• deploy/controller.yaml's third container, csi-snapshotter, and deploy/controller-rbac.yaml's second ClusterRole, both trimmed from real upstream reference manifests the same honest way Chapter 7 trimmed external-provisioner's
• deploy/volumesnapshotclass.yaml, and the first cluster-level, not-driver-specific component this book has ever had to install: snapshot-controller, shared infrastructure any CSI driver's snapshot support depends on
• Two forms of proof again, grpcurl by hand and a real VolumeSnapshot object watched end to end — data written through a real mount, copied by a real CreateSnapshot call nothing typed by hand ever triggered, and readable straight off the node's disk afterward

Two honest gaps remain, both already named directly above: ListSnapshots stays an unadvertised, correctly-skipped stub, and CreateVolume still can't restore a volume from a snapshot's contents — volume_content_source sits in CreateVolumeRequest, unread, same as every chapter since Chapter 7. Both stay that way for the rest of this book — deliberate, not deferred. Chapter 11 turns to a different kind of gap: not a missing RPC, but everything this driver is still missing to be safely operated for real — structured error codes, logs, and metrics that can actually be scraped off a running pod.

Chapter 11: Getting to Production

A fire drill and a real fire look identical for the first ten seconds. The alarm sounds the same. People stand up the same way. What tells them apart is everything that was already in place before the alarm went off: whether the exits were marked, whether anyone knew which stairwell to use, whether the building even had a plan. Production incidents work the same way. The gRPC call that fails at 3am looks exactly like the gRPC call that failed in a unit test at 3pm — same error, same stack, same shape. What tells them apart is everything already in place before that call failed: whether the error said anything useful, whether anyone could see it happening, whether a retry made things better or quietly made them worse.

localdir-csi works. Every RPC this book has built is tested, green, and proven against a real cluster. That was never really in question — TDD makes "does it work" almost boring to answer. What this chapter asks is a different question: if this driver were running for real, at 3am, with nobody watching, would anyone be able to tell what went wrong? Right now, mostly not. This chapter closes that gap in two concrete ways — structured, auditable error codes, and logs and metrics an operator can actually use, actually scraped off a real running pod — without changing what this driver does, only what it's like to run.

What "production" doesn't mean here

Worth saying plainly, the way this book has named every other honest simplification: this chapter does not turn localdir-csi into something you'd actually run in a real cluster. It still stores volumes as plain directories on one node, with no real capacity enforcement, no multi-node story, and no real backend underneath it. Volume Expansion was cut from this book's plan for exactly this reason — ControllerExpandVolume/NodeExpandVolume would have nothing real to exercise, since CreateVolume never enforced a real size limit to begin with. Padding out an RPC nobody could meaningfully test isn't "getting to production," it's decoration.

What is real, and worth a whole chapter: the operational concerns that apply to this driver exactly as much as they'd apply to a driver backed by a real SAN. Bad error codes confuse real callers regardless of what's underneath. Missing logs and metrics leave a real operator blind regardless of what's underneath. Both problems are real, and fixable, without pretending this toy driver is something it isn't.

Structured error codes: an audit, and a DRY pass

Every RPC in this driver already returns a codes.X — that's been true since Chapter 3. What hasn't happened yet is checking that every one of those codes actually matches what the CSI spec says callers should expect. The spec's own spec.md documents, RPC by RPC, which conditions map to which gRPC code — the same kind of upstream source this book has checked before drafting since Chapter 3's "check the interfaces first" convention, just applied retroactively this time, against code that already shipped.

Read internal/driver/controller.go and internal/driver/node.go against that table, RPC by RPC, and it holds up. A few gaps are real but already-named simplifications — DeleteVolume never returns FAILED_PRECONDITION for "volume in use," because this driver has no real attachment tracking to know that; ControllerPublishVolume never returns NOT_FOUND for a nonexistent node, because a single-node driver has no node registry to check against. Both already flagged, in Chapter 7 and Chapter 8's own text — nothing new to fix there.

Two other gaps used to live on this list too: CreateVolume not checking capacity on retry, and NodePublishVolume not checking access mode on retry. Both real bugs, not simplifications — and both already closed by the time this chapter starts. NodePublishVolume got its fix in Chapter 6, the moment its own IsMountPoint check turned out to answer a narrower question than "is this genuinely the same request." CreateVolume got its fix in Chapter 9, the moment a real csi-sanity run turned the same suspicion into a real failing test. Nothing left to fix here — but the audit does turn up something the fixes themselves left behind.

Read those two fixes next to CreateSnapshot's own idempotency check from Chapter 10, and the same nine lines show up in all three, changed only in which metadata type they read and what they compare:

if existing, err := readXMeta(d.dataDir, id); err == nil {
	if <conflict> {
		return nil, status.Errorf(codes.AlreadyExists, ...)
	}
	// ...use existing, return early...
} else if !os.IsNotExist(err) {
	return nil, status.Errorf(codes.Internal, "reading ... metadata: %v", err)
}

Three RPCs, three metadata types, one repeated shape: read a record, and turn "does it exist" into one of three answers — no record yet, here's the record, or something went wrong reading it. Always DRY says that repetition is overdue for collapsing, the same way Chapter 6's own test helpers collapsed a different repeated shape a few chapters back.

Add internal/driver/meta.go:

package driver

import "os"

// readMetaOrZero reads metadata via read. If a record exists, it comes
// back with found=true. If none exists yet, readMetaOrZero returns the
// zero value with found=false and no error — the one signal every
// idempotency check in this driver needs before deciding whether a
// request is the first one for this name, or a retry of one already
// recorded. Any other read error is returned as-is, for the caller to
// wrap into its own codes.Internal message.
func readMetaOrZero[T any](read func() (T, error)) (meta T, found bool, err error) {
	meta, err = read()
	if err == nil {
		return meta, true, nil
	}
	if os.IsNotExist(err) {
		var zero T
		return zero, false, nil
	}
	var zero T
	return zero, false, err
}

Then each of the three call sites shrinks by a line and reads a little straighter. CreateVolume, in internal/driver/controller.go:

	existing, found, err := readMetaOrZero(func() (volumeMeta, error) {
		return readVolumeMeta(d.dataDir, req.GetName())
	})
	if err != nil {
		return nil, status.Errorf(codes.Internal, "reading volume %q metadata: %v", req.GetName(), err)
	}
	if found {
		if existing.CapacityBytes != requestedBytes {
			return nil, status.Errorf(codes.AlreadyExists, "volume %q already exists with a different capacity", req.GetName())
		}
		return &csi.CreateVolumeResponse{
			Volume: &csi.Volume{
				VolumeId:      req.GetName(),
				CapacityBytes: existing.CapacityBytes,
			},
		}, nil
	}

CreateSnapshot, same file:

	existingSnap, foundSnap, err := readMetaOrZero(func() (snapshotMeta, error) {
		return readSnapshotMeta(d.dataDir, req.GetName())
	})
	if err != nil {
		return nil, status.Errorf(codes.Internal, "reading snapshot %q metadata: %v", req.GetName(), err)
	}
	if foundSnap {
		if existingSnap.SourceVolumeID != req.GetSourceVolumeId() {
			return nil, status.Errorf(codes.AlreadyExists, "snapshot %q already exists for a different source volume", req.GetName())
		}
		return &csi.CreateSnapshotResponse{
			Snapshot: &csi.Snapshot{
				SnapshotId:     req.GetName(),
				SourceVolumeId: existingSnap.SourceVolumeID,
				SizeBytes:      existingSnap.SizeBytes,
				CreationTime:   timestamppb.New(existingSnap.CreationTime),
				ReadyToUse:     true,
			},
		}, nil
	}

NodePublishVolume, in internal/driver/node.go — the variable is renamed metaErr here only because err already names the result of the IsMountPoint call one line up:

	if mounted {
		existing, found, metaErr := readMetaOrZero(func() (nodePublishMeta, error) {
			return readNodePublishMeta(d.dataDir, req.GetVolumeId())
		})
		if metaErr != nil {
			return nil, status.Errorf(codes.Internal, "reading volume %q publish metadata: %v", req.GetVolumeId(), metaErr)
		}
		if found && (existing.TargetPath != req.GetTargetPath() || existing.Readonly != req.GetReadonly()) {
			return nil, status.Errorf(codes.AlreadyExists, "volume %q is already published with different parameters", req.GetVolumeId())
		}
		return &csi.NodePublishVolumeResponse{}, nil
	}

This refactor doesn't earn a new red test — nothing about what these three RPCs do changes, only how they ask the same question of their own metadata. The 46 tests already in the suite, spanning every RPC this book has built so far, are the safety net:

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.013s

Green, unchanged. Three call sites, one shared shape, one file (meta.go) that didn't exist an hour ago — Always DRY, applied to code this book already trusted, not just code it was about to write.

Capability enforcement: a gap the audit above missed

The error-code audit above checked what each RPC returns. It didn't check what each RPC accepts — and that gap is real, not hypothetical. ValidateVolumeCapabilities (Chapter 9) already knows exactly what this driver supports: a mount volume, never a raw block volume, requesting SINGLE_NODE_WRITER access. But ValidateVolumeCapabilities is an advisory RPC — nothing forces a caller to invoke it before CreateVolume, ControllerPublishVolume, or NodePublishVolume, and none of those three actually checks the capability it was handed against that same rule. Right now, a CreateVolume call naming a block-volume capability succeeds anyway; a NodePublishVolume call naming MULTI_NODE_MULTI_WRITER still gets a real bind mount. ValidateVolumeCapabilities would have correctly said no to both — it just never gets asked.

Always DRY says the check belongs in one place, not copy-pasted into four RPCs with four chances to drift out of sync with each other. Add internal/driver/capabilities.go:

package driver

import "github.com/container-storage-interface/spec/lib/go/csi"

// supportsVolumeCapability reports whether c is a capability this
// driver can actually honor: a mount volume — never a raw block
// volume — requesting SINGLE_NODE_WRITER access, the only combination
// Chapter 9's csi-sanity run ever exercised successfully. Every RPC
// that receives a *csi.VolumeCapability calls this before doing any
// real work with it, not just ValidateVolumeCapabilities, whose whole
// job is answering this exact question.
func supportsVolumeCapability(c *csi.VolumeCapability) bool {
	return c.GetMount() != nil &&
		c.GetAccessMode().GetMode() == csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER
}

With a test, internal/driver/capabilities_test.go:

package driver

import (
	"testing"

	"github.com/container-storage-interface/spec/lib/go/csi"
)

func TestSupportsVolumeCapability_AcceptsSupported(t *testing.T) {
	c := &csi.VolumeCapability{
		AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
		AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER},
	}
	if !supportsVolumeCapability(c) {
		t.Error("supportsVolumeCapability() = false, want true for mount + SINGLE_NODE_WRITER")
	}
}

func TestSupportsVolumeCapability_RejectsUnsupported(t *testing.T) {
	cases := map[string]*csi.VolumeCapability{
		"block volume": {
			AccessType: &csi.VolumeCapability_Block{Block: &csi.VolumeCapability_BlockVolume{}},
			AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER},
		},
		"multi-node access mode": {
			AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
			AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER},
		},
	}
	for name, c := range cases {
		if supportsVolumeCapability(c) {
			t.Errorf("supportsVolumeCapability(%s) = true, want false", name)
		}
	}
}

ValidateVolumeCapabilities's own capability loop (Chapter 9) already checked exactly this, just inline and in two separate if statements. Collapse it onto the new helper:

	for _, c := range req.GetVolumeCapabilities() {
		if !supportsVolumeCapability(c) {
			return &csi.ValidateVolumeCapabilitiesResponse{
				Message: "capability not supported: only a mount volume with SINGLE_NODE_WRITER access is supported",
			}, nil
		}
	}

Then the three RPCs that were missing the check entirely. In CreateVolume (internal/driver/controller.go), right after the existing "capabilities are required" presence check:

	for _, c := range req.GetVolumeCapabilities() {
		if !supportsVolumeCapability(c) {
			return nil, status.Errorf(codes.InvalidArgument, "unsupported volume_capability: only a mount volume with SINGLE_NODE_WRITER access is supported")
		}
	}

In ControllerPublishVolume, same file, right after its own "volume_capability is required" check:

	if !supportsVolumeCapability(req.GetVolumeCapability()) {
		return nil, status.Error(codes.InvalidArgument, "unsupported volume_capability: only a mount volume with SINGLE_NODE_WRITER access is supported")
	}

And in NodePublishVolume (internal/driver/node.go), right after its own "volume_capability is required" check:

	if !supportsVolumeCapability(req.GetVolumeCapability()) {
		return nil, status.Error(codes.InvalidArgument, "unsupported volume_capability: only a mount volume with SINGLE_NODE_WRITER access is supported")
	}

Two tests prove the wiring, not just the helper. In internal/driver/controller_test.go:

func TestCreateVolume_RejectsUnsupportedCapability(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.CreateVolumeRequest{
		Name: "vol-1",
		VolumeCapabilities: []*csi.VolumeCapability{{
			AccessType: &csi.VolumeCapability_Block{Block: &csi.VolumeCapability_BlockVolume{}},
			AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER},
		}},
	}

	_, err := d.CreateVolume(t.Context(), req)
	requireStatusCode(t, err, codes.InvalidArgument, req)
}

And in internal/driver/node_test.go:

func TestNodePublishVolume_RejectsUnsupportedCapability(t *testing.T) {
	d := newTestDriver(t, t.TempDir(), &fakeMounter{})

	req := &csi.NodePublishVolumeRequest{
		VolumeId:   "vol-1",
		TargetPath: filepath.Join(t.TempDir(), "target"),
		VolumeCapability: &csi.VolumeCapability{
			AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
			AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER},
		},
	}

	_, err := d.NodePublishVolume(t.Context(), req)
	requireStatusCode(t, err, codes.InvalidArgument, req)
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.014s

Green — four RPCs now agree on what this driver supports, checked in exactly one place, instead of one RPC knowing the rule and three others silently ignoring it.

Path safety: an audit of everything this driver joins onto a path

One more gap the error-code audit didn't cover, because it's not about codes at all: every RPC that touches the filesystem builds a path by joining d.dataDir with a caller-supplied string — a volume ID, a snapshot ID — and CSI identifiers are opaque by spec. Nothing stops a caller from naming a volume ../../etc or snapshots (this driver's own reserved subdirectory). The external-provisioner-generated names this book's own proofs have used so far happen to look safe; the spec never promised that, and every grpcurl demonstration since Chapter 7 has reached this driver's socket directly, no sidecar validation in front of it at all.

One helper closes every one of those joins at once, in internal/driver/paths.go:

package driver

import (
	"fmt"
	"path/filepath"
	"strings"
)

// reservedNames are subdirectories this driver's own bookkeeping
// already owns under dataDir. A volume or snapshot ID that collided
// with one of these would silently corrupt the driver's own metadata
// instead of failing loudly.
var reservedNames = map[string]bool{
	"snapshots": true,
	".csi-meta": true,
}

// safeChildPath joins name onto root, the same way every RPC in this
// driver already has with filepath.Join — but refuses to do it at all
// if name could escape root or collide with a reserved directory.
// CSI volume/snapshot IDs are caller-supplied and opaque; this driver
// has no sidecar in front of it once a caller reaches its socket
// directly, so it has to be the one place that says no.
func safeChildPath(root, name string) (string, error) {
	if name == "" {
		return "", fmt.Errorf("name must not be empty")
	}
	if reservedNames[name] {
		return "", fmt.Errorf("name %q collides with a reserved internal directory", name)
	}
	if name == "." || name == ".." || strings.ContainsAny(name, `/\`) {
		return "", fmt.Errorf("name %q is not a valid path component", name)
	}
	child := filepath.Join(root, name)
	cleanRoot := filepath.Clean(root)
	if child != cleanRoot && !strings.HasPrefix(child, cleanRoot+string(filepath.Separator)) {
		return "", fmt.Errorf("name %q escapes the data root", name)
	}
	return child, nil
}

The separator-and-..-literal checks catch the common case in plain English; the final prefix check is the one that actually matters, because filepath.Join already runs filepath.Clean internally — anything cleverer than a literal .. component that still resolves outside root after cleaning gets caught there, not by the earlier checks. Four cases worth a real test each, in internal/driver/paths_test.go:

package driver

import (
	"path/filepath"
	"testing"
)

func TestSafeChildPath_AcceptsValidName(t *testing.T) {
	root := t.TempDir()
	got, err := safeChildPath(root, "vol-1")
	if err != nil {
		t.Fatalf("safeChildPath() error = %v, want nil", err)
	}
	if want := filepath.Join(root, "vol-1"); got != want {
		t.Errorf("safeChildPath() = %q, want %q", got, want)
	}
}

func TestSafeChildPath_RejectsTraversal(t *testing.T) {
	if _, err := safeChildPath(t.TempDir(), "../escape"); err == nil {
		t.Error("safeChildPath() = nil error, want one for a traversal attempt")
	}
}

func TestSafeChildPath_RejectsSeparators(t *testing.T) {
	if _, err := safeChildPath(t.TempDir(), "sub/dir"); err == nil {
		t.Error("safeChildPath() = nil error, want one for an embedded separator")
	}
}

func TestSafeChildPath_RejectsReservedNames(t *testing.T) {
	root := t.TempDir()
	for _, name := range []string{"snapshots", ".csi-meta"} {
		if _, err := safeChildPath(root, name); err == nil {
			t.Errorf("safeChildPath(%q) = nil error, want one for a reserved name", name)
		}
	}
}

Wiring it in means every existing filepath.Join(d.dataDir, ...) (or filepath.Join(dataDir, ...) inside a metadata-path helper) becomes a checked call instead, returning codes.InvalidArgument on rejection — an invalid ID is a caller mistake, not a server failure. The call sites, each a one-line change plus an error check:

Each replacement follows the same shape:

	path, err := safeChildPath(d.dataDir, req.GetVolumeId())
	if err != nil {
		return nil, status.Errorf(codes.InvalidArgument, "invalid volume_id: %v", err)
	}

— path used exactly where the old filepath.Join result was, nothing else in any of these methods changes. Run the full suite after wiring every site in the table above:

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.015s

Green — every id this driver ever joins onto a real path now goes through one function that can say no, and that function has its own tests proving it actually does.

Structured logging: one interceptor, every RPC

The second problem named above: missing logs leave a real operator blind. Right now, nothing in internal/driver logs anything. A CreateVolume call that fails shows up as a gRPC error the caller sees — and nowhere else. On a real cluster, the caller is usually external-provisioner, several hops away from whoever is debugging.

The tempting fix is to add a klog.InfoS call at the top and bottom of every RPC method. Always DRY says no: that's the same four lines, copy-pasted into eleven methods, all of them needing to change in lockstep the day the log format changes. gRPC has a purpose-built answer to exactly this shape of problem — a unary interceptor: a function that wraps every RPC call, given the request, the handler, and a chance to run code before and after it runs. Wire one in once, and every RPC — this one and every one added later — gets logging for free, without a single klog call inside any RPC method itself. This is Open/Closed in its purest form: adding logging to RPC number twelve requires touching zero existing files.

k8s.io/klog/v2 is the structured logging package the rest of Kubernetes uses — InfoS and ErrorS take a message and alternating key/value pairs, rather than a Printf-style format string. Add it the same way every other real dependency in this book got added, back in Chapter 2:

go get k8s.io/klog/v2

klog has no public Output variable to redirect for a test — capturing its output means installing a real logr.Logger via SetLogger, the same way klog's own real test suite does it. k8s.io/klog/v2/textlogger supplies a ready-made one: a logr.Logger that writes plain text to whatever io.Writer it's given, and it's already part of the k8s.io/klog/v2 module — no separate dependency to add for it.

Start with the test, in internal/driver/logging_test.go:

package driver

import (
	"bytes"
	"context"
	"errors"
	"strings"
	"testing"

	"google.golang.org/grpc"
	"k8s.io/klog/v2"
	"k8s.io/klog/v2/textlogger"
)

func TestLoggingInterceptor_LogsSuccess(t *testing.T) {
	var buf bytes.Buffer
	klog.SetLogger(textlogger.NewLogger(textlogger.NewConfig(textlogger.Output(&buf))))
	t.Cleanup(klog.ClearLogger)

	handler := func(ctx context.Context, req any) (any, error) {
		return "ok", nil
	}
	info := &grpc.UnaryServerInfo{FullMethod: "/csi.v1.Controller/CreateVolume"}

	_, err := LoggingInterceptor(t.Context(), nil, info, handler)
	if err != nil {
		t.Fatalf("LoggingInterceptor() returned an error: %v", err)
	}

	got := buf.String()
	if !strings.Contains(got, "RPC succeeded") {
		t.Errorf("log output = %q, want it to contain %q", got, "RPC succeeded")
	}
	if !strings.Contains(got, "/csi.v1.Controller/CreateVolume") {
		t.Errorf("log output = %q, want it to contain the method name", got)
	}
}

func TestLoggingInterceptor_LogsFailure(t *testing.T) {
	var buf bytes.Buffer
	klog.SetLogger(textlogger.NewLogger(textlogger.NewConfig(textlogger.Output(&buf))))
	t.Cleanup(klog.ClearLogger)

	wantErr := errors.New("volume not found")
	handler := func(ctx context.Context, req any) (any, error) {
		return nil, wantErr
	}
	info := &grpc.UnaryServerInfo{FullMethod: "/csi.v1.Controller/DeleteVolume"}

	_, err := LoggingInterceptor(t.Context(), nil, info, handler)
	if err != wantErr {
		t.Fatalf("LoggingInterceptor() error = %v, want %v", err, wantErr)
	}

	got := buf.String()
	if !strings.Contains(got, "RPC failed") {
		t.Errorf("log output = %q, want it to contain %q", got, "RPC failed")
	}
	if !strings.Contains(got, "volume not found") {
		t.Errorf("log output = %q, want it to contain the error", got)
	}
}

Each test installs a textlogger pointed at a bytes.Buffer instead of the default stderr, and clears it in t.Cleanup via klog.ClearLogger, so no other test in the package inherits a logger whose buffer has since gone out of scope. klog.InfoS/ErrorS route through whatever's installed with SetLogger — install a logger, and every top-level klog.InfoS/ErrorS call anywhere in the program starts going through it instead of the classic stderr path.

Neither LoggingInterceptor nor grpc.UnaryServerInfo/UnaryHandler exist yet:

go test ./internal/driver/... -run TestLoggingInterceptor

./logging_test.go:25:12: undefined: LoggingInterceptor
./logging_test.go:50:12: undefined: LoggingInterceptor
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]

grpc.UnaryServerInfo, grpc.UnaryHandler, and grpc.UnaryServerInterceptor are real types this driver's existing google.golang.org/grpc dependency already provides — no new module to add for them.

With those in place, internal/driver/logging.go:

package driver

import (
	"context"
	"time"

	"google.golang.org/grpc"
	"k8s.io/klog/v2"
)

// LoggingInterceptor logs every unary RPC call once, on the way out:
// a structured success line with the method name and how long it took,
// or a structured error line with the method name and the error. Wire
// it in once, as a grpc.UnaryInterceptor, and every RPC — this one and
// every one added later — gets consistent logging for free, without a
// single klog call inside any RPC method itself.
func LoggingInterceptor(
	ctx context.Context,
	req any,
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (any, error) {
	start := time.Now()
	resp, err := handler(ctx, req)
	duration := time.Since(start)

	if err != nil {
		klog.ErrorS(err, "RPC failed", "method", info.FullMethod, "duration", duration)
		return resp, err
	}

	klog.InfoS("RPC succeeded", "method", info.FullMethod, "duration", duration)
	return resp, nil
}

go test ./internal/driver/... -run TestLoggingInterceptor -v

=== RUN   TestLoggingInterceptor_LogsSuccess
--- PASS: TestLoggingInterceptor_LogsSuccess (0.00s)
=== RUN   TestLoggingInterceptor_LogsFailure
--- PASS: TestLoggingInterceptor_LogsFailure (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green. LoggingInterceptor is a plain function — nothing about it requires a running server, which is exactly why it could be unit tested by calling it directly with a stub handler, the same way every RPC method in this book has been tested since Chapter 3.

It's worth being precise about what actually happens at request time, because LoggingInterceptor's signature — (ctx, req, info, handler) — looks like an ordinary function until you notice that one of its arguments is itself a function it hasn't called yet. When a CreateVolume request arrives at a server wired with grpc.UnaryInterceptor(driver.LoggingInterceptor), gRPC's own machinery decodes the request off the wire into a *csi.CreateVolumeRequest, and then — instead of calling the driver's CreateVolume method directly — packages that call up as a closure (a grpc.UnaryHandler value, func(ctx, req) (any, error)) and calls LoggingInterceptor(ctx, req, info, thatClosure). The interceptor is handed "the rest of this request's life" as a value; nothing runs until the interceptor's own body decides to run it.

From there, LoggingInterceptor's four lines run in an order worth tracing explicitly:

1. start := time.Now() runs first — genuinely before the RPC, code the interceptor controls and CreateVolume never sees.
2. handler(ctx, req) — the interceptor calls the closure it was handed. This is the moment CreateVolume actually runs. Everything inside it is invisible to LoggingInterceptor, which only sees what goes in and what eventually comes back out.
3. Control returns to LoggingInterceptor, now holding resp, err, and duration := time.Since(start) — all three exist only because the interceptor waited for handler to return before computing anything with them.
4. klog.InfoS/ErrorS runs — genuinely after the RPC — and LoggingInterceptor returns (resp, err) unchanged, back up to gRPC, which encodes it onto the wire exactly as if no interceptor existed at all.

That's the whole mechanism: an interceptor is a function willing to call another function partway through its own body, with license to run code on either side of that call. CreateVolume never sees LoggingInterceptor and never needs to — Chapter 6's Dependency Inversion note (depend on abstractions, not concretions) shows up here in a different shape: CreateVolume doesn't depend on an abstraction for logging, it depends on nothing at all, because logging was never its job to know about in the first place.

Wiring it in is one more line in cmd/localdir-csi/main.go, alongside the existing service registrations:

	server := grpc.NewServer(grpc.UnaryInterceptor(driver.LoggingInterceptor))
	csi.RegisterIdentityServer(server, d)
	csi.RegisterControllerServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

Every RPC this driver has, or will ever have, now logs a structured success or failure line — without one line of logging code inside CreateVolume, NodePublishVolume, or any of the other nine methods already written.

Metrics: the same pattern, twice

Missing logs is half the observability gap named earlier — the other half is missing metrics. Logs answer "what happened on this one call"; metrics answer "how is this driver doing, in aggregate, right now" — how many CreateVolume calls failed in the last five minutes, what NodePublishVolume's p99 latency looks like. A real operator needs both, and neither one substitutes for the other.

The DRY argument from the logging section applies again, unchanged: instrumenting every RPC method by hand means eleven near-identical blocks of counter and histogram code, all needing to move together. MetricsInterceptor is a second, independent interceptor, built the same way as LoggingInterceptor — which is worth pausing on, because it's a small proof of Single Responsibility working as advertised. Logging and metrics are two different concerns; the interceptor pattern lets them live in two different files, each one just as unit-testable in isolation as the other, neither one aware the other exists.

github.com/prometheus/client_golang/prometheus is the real metrics library most of the Kubernetes ecosystem uses. Two shapes matter here. A CounterVec is a family of monotonically-increasing counts, one per distinct combination of label values — rpc_requests_total{method="...",code="OK"} is a different counter from rpc_requests_total{method="...",code="NotFound"}, even though both live under the same CounterVec. A HistogramVec is the same idea applied to latency: instead of one number, each label combination gets a set of cumulative bucket counts (Buckets — how many observations fell at or below each threshold) plus a running sum and count, which is what lets a real Prometheus server compute p50/p95/p99 later without this driver ever calculating a percentile itself. Both are built from an Opts struct (CounterOpts/HistogramOpts) whose Namespace/Subsystem/Name fields get joined with underscores into the final metric name — this driver's Subsystem: "localdir_csi", Name: "rpc_requests_total" becomes localdir_csi_rpc_requests_total on the wire.

"On the wire" is worth making concrete, because this chapter is about to build it for real, against a real running pod: once a driver MustRegisters its collectors and serves promhttp.Handler() on an HTTP endpoint (a real but separate piece of plumbing — this driver's Unix-socket gRPC server has no HTTP port of its own), a real Prometheus server scraping it sees plain text shaped like this:

# HELP localdir_csi_rpc_requests_total Total number of RPCs handled, by method and result code.
# TYPE localdir_csi_rpc_requests_total counter
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Controller/CreateVolume"} 12
localdir_csi_rpc_requests_total{code="NotFound",method="/csi.v1.Controller/DeleteVolume"} 1

# HELP localdir_csi_rpc_duration_seconds RPC latency in seconds, by method.
# TYPE localdir_csi_rpc_duration_seconds histogram
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.005"} 0
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.01"} 3
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.025"} 9
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="+Inf"} 12
localdir_csi_rpc_duration_seconds_sum{method="/csi.v1.Controller/CreateVolume"} 0.183
localdir_csi_rpc_duration_seconds_count{method="/csi.v1.Controller/CreateVolume"} 12

Every le ("less than or equal") bucket is cumulative — the +Inf bucket always equals the total count — which is what lets Prometheus interpolate a percentile across buckets it never saw an individual observation land in directly. None of this book's code produces that text; real client_golang does, via its expfmt encoder, the moment something calls promhttp.Handler().

Add the real library the same way, one go get for the package and its test-only testutil helper:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/testutil
go get github.com/prometheus/client_model/go

A metric's value can only be read back out two ways: scrape /metrics over HTTP for real, or — inside a test, without a server — call github.com/prometheus/client_golang/prometheus/testutil.ToFloat64, a genuine public export built for exactly this. It takes any prometheus.Collector that reports exactly one number and returns that number; CounterVec.WithLabelValues(...) returns a Counter, and the real Counter interface already embeds Collector, so handing one straight to ToFloat64 needs no extra plumbing.

Histograms are a different story — ToFloat64 explicitly panics on anything that isn't a Gauge, Counter, or Untyped, because a histogram doesn't reduce to one number. The real, idiomatic way to check what a histogram recorded — the same pattern client_golang's own test suite uses — is to Write it into a dto.Metric (the same protobuf message a real scrape produces) and read GetHistogram().GetSampleCount() off that. One wrinkle worth knowing before it surprises a reader: HistogramVec.WithLabelValues(...) returns the real API's narrow Observer interface (just Observe(float64)), not the fuller Histogram interface that has Write on it — reaching Write needs an explicit type assertion, s.(prometheus.Histogram), the same way real code calling this must.

The test, in internal/driver/metrics_test.go:

package driver

import (
	"context"
	"testing"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/testutil"
	dto "github.com/prometheus/client_model/go"
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
)

func TestMetricsInterceptor_RecordsSuccess(t *testing.T) {
	handler := func(ctx context.Context, req any) (any, error) { return "ok", nil }
	info := &grpc.UnaryServerInfo{FullMethod: "/csi.v1.Controller/CreateVolume"}

	before := testutil.ToFloat64(rpcRequestsTotal.WithLabelValues(info.FullMethod, "OK"))

	if _, err := MetricsInterceptor(t.Context(), nil, info, handler); err != nil {
		t.Fatalf("MetricsInterceptor() returned an error: %v", err)
	}

	after := testutil.ToFloat64(rpcRequestsTotal.WithLabelValues(info.FullMethod, "OK"))
	if after != before+1 {
		t.Errorf("rpc_requests_total{method=%q,code=OK} = %v, want %v", info.FullMethod, after, before+1)
	}

	var m dto.Metric
	if err := rpcDurationSeconds.WithLabelValues(info.FullMethod).(prometheus.Histogram).Write(&m); err != nil {
		t.Fatalf("Write() failed: %v", err)
	}
	if m.GetHistogram().GetSampleCount() == 0 {
		t.Errorf("rpc_duration_seconds{method=%q} recorded no observations", info.FullMethod)
	}
}

func TestMetricsInterceptor_RecordsFailure(t *testing.T) {
	handler := func(ctx context.Context, req any) (any, error) {
		return nil, status.Error(codes.NotFound, "volume not found")
	}
	info := &grpc.UnaryServerInfo{FullMethod: "/csi.v1.Controller/DeleteVolume"}

	before := testutil.ToFloat64(rpcRequestsTotal.WithLabelValues(info.FullMethod, "NotFound"))

	if _, err := MetricsInterceptor(t.Context(), nil, info, handler); err == nil {
		t.Fatal("MetricsInterceptor() returned no error, want NotFound")
	}

	after := testutil.ToFloat64(rpcRequestsTotal.WithLabelValues(info.FullMethod, "NotFound"))
	if after != before+1 {
		t.Errorf("rpc_requests_total{method=%q,code=NotFound} = %v, want %v", info.FullMethod, after, before+1)
	}
}

Both tests read before/after deltas on the package-level counters rather than asserting an absolute value — rpcRequestsTotal is shared across every test in the package, so any test asserting an absolute count would be quietly coupled to every other test's call order. The histogram check only appears in the success test, and only checks that something was recorded, not an absolute count, for the same reason.

go test ./internal/driver/... -run TestMetricsInterceptor

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
internal/driver/metrics_test.go:7:2: "github.com/prometheus/client_golang/prometheus" imported and not used
internal/driver/metrics_test.go:19:31: undefined: rpcRequestsTotal
internal/driver/metrics_test.go:21:15: undefined: MetricsInterceptor
internal/driver/metrics_test.go:25:30: undefined: rpcRequestsTotal
internal/driver/metrics_test.go:31:12: undefined: rpcDurationSeconds
internal/driver/metrics_test.go:45:31: undefined: rpcRequestsTotal
internal/driver/metrics_test.go:47:15: undefined: MetricsInterceptor
internal/driver/metrics_test.go:51:30: undefined: rpcRequestsTotal
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]

That first line is real, and a little counterintuitive: prometheus genuinely is used, at the .(prometheus.Histogram) assertion further down — but with rpcDurationSeconds undefined, the compiler's error recovery gives up on that whole statement before it reaches the assertion, so it never marks the import as referenced. It's a real Go compiler quirk under cascading errors, not a mistake in the test.

internal/driver/metrics.go:

package driver

import (
	"context"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"google.golang.org/grpc"
	"google.golang.org/grpc/status"
)

var (
	rpcRequestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: "localdir_csi",
			Name:      "rpc_requests_total",
			Help:      "Total number of RPCs handled, by method and result code.",
		},
		[]string{"method", "code"},
	)

	rpcDurationSeconds = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Subsystem: "localdir_csi",
			Name:      "rpc_duration_seconds",
			Help:      "RPC latency in seconds, by method.",
			Buckets:   []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
		},
		[]string{"method"},
	)
)

// MetricsInterceptor records one counter increment and one duration
// observation per RPC call, labeled by method and (for the counter) the
// result code. Wire it in alongside LoggingInterceptor — the two are
// independent, so either can be added, removed, or reused on its own.
func MetricsInterceptor(
	ctx context.Context,
	req any,
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (any, error) {
	start := time.Now()
	resp, err := handler(ctx, req)
	duration := time.Since(start).Seconds()

	rpcRequestsTotal.WithLabelValues(info.FullMethod, status.Code(err).String()).Inc()
	rpcDurationSeconds.WithLabelValues(info.FullMethod).Observe(duration)

	return resp, err
}

go test ./internal/driver/... -run TestMetricsInterceptor -v

=== RUN   TestMetricsInterceptor_RecordsSuccess
--- PASS: TestMetricsInterceptor_RecordsSuccess (0.00s)
=== RUN   TestMetricsInterceptor_RecordsFailure
--- PASS: TestMetricsInterceptor_RecordsFailure (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.006s

Green — two interceptors now, LoggingInterceptor and MetricsInterceptor, each tested in complete isolation from the other. The wiring from the logging section only has room for one: grpc.UnaryInterceptor installs exactly one interceptor. The real grpc package's answer for more than one is grpc.ChainUnaryInterceptor, which takes any number of interceptors and nests them: the first one listed ends up outermost, wrapping every interceptor after it the same way nested function calls wrap each other — first's code runs before second's on the way in, and after second's on the way back out. That matters for this driver's wiring, because it decides what a caller sees timed:

	server := grpc.NewServer(grpc.ChainUnaryInterceptor(
		driver.LoggingInterceptor,
		driver.MetricsInterceptor,
	))
	csi.RegisterIdentityServer(server, d)
	csi.RegisterControllerServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

LoggingInterceptor listed first means its own time.Since(start) includes MetricsInterceptor's work — the log line reports how long the RPC actually took a caller to get an answer, metrics recording included, not just the bare handler.

Serving metrics for real

Recording a metric and being able to see it are two different things. rpcRequestsTotal and rpcDurationSeconds update in memory on every RPC — that part is already proven by metrics_test.go. But nothing so far reads those numbers back out except a unit test calling testutil.ToFloat64 directly. On a real cluster, the thing that reads them out is a Prometheus server, and it can only do that by making an HTTP request. This driver doesn't have an HTTP server — its only listener is the Unix-socket gRPC server kubelet talks to. Fixing that is a few real lines, not an exercise, so this section builds it and then goes and looks at it running on a real pod.

CounterVec/HistogramVec values only show up on a scrape once they're registered with a prometheus.Registry — creating them with NewCounterVec/NewHistogramVec does not register them anywhere on its own. prometheus.MustRegister does that; it panics on a registration error (a name collision, say), which is the right failure mode here — a metrics wiring bug should be loud at startup, not silently drop data forever. Add the registration call to internal/driver/metrics.go, right after the two var declarations:

func init() {
	prometheus.MustRegister(rpcRequestsTotal, rpcDurationSeconds)
}

prometheus.MustRegister with no explicit Registry argument registers against prometheus.DefaultRegisterer — the same default registry promhttp.Handler() reads from with no arguments of its own. That pairing is why the wiring in main.go doesn't have to pass anything explicit between the two: registering here and serving there agree on "the default registry" without either line naming it.

main.go gets one more piece, alongside the existing gRPC server startup — a second server, on a second port, running the whole time the driver runs. Two new imports: net/http, the standard library's own HTTP server, and github.com/prometheus/client_golang/prometheus/promhttp, the sub-package that turns a Registry into an http.Handler:

	go func() {
		http.Handle("/metrics", promhttp.Handler())
		if err := http.ListenAndServe(":9090", nil); err != nil {
			klog.ErrorS(err, "metrics server failed")
		}
	}()

That go func() matters as much as the two lines inside it. http.ListenAndServe blocks — it doesn't return until the server stops or fails — so without go, starting the metrics server would mean the driver's real job, serving the CSI gRPC socket, never starts. Running it in its own goroutine means the metrics HTTP server and the gRPC server run side by side, on two different listeners (:9090 for HTTP, the Unix socket for gRPC), each blocking only its own goroutine.

The deploy manifest needs to say the port exists, or nothing outside the pod's own network namespace can reach it. Both deploy/node.yaml and deploy/controller.yaml run this same driver binary, so both containers get the same addition:

        ports:
        - containerPort: 9090
          name: metrics

That's everything the code needs. The rest of this section is running it for real: rebuild and redeploy the driver, then reach port 9090 on a live pod the same way kubectl port-forward reaches any other pod port, and see what a real Prometheus scrape would see.

make deploy
kubectl port-forward deployment/localdir-csi-controller 9090:9090

make deploy already waits for both the Controller and Node rollouts to finish (Chapter 2), so by the time port-forward runs, the pod on the other end is genuinely the freshly built binary, not whatever was running before.

Then, in a second terminal, with a couple of volumes already created against the driver so the counters aren't sitting at zero:

curl -s localhost:9090/metrics | grep localdir_csi

# HELP localdir_csi_rpc_duration_seconds RPC latency in seconds, by method.
# TYPE localdir_csi_rpc_duration_seconds histogram
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.005"} 23
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.01"} 23
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="0.025"} 23
localdir_csi_rpc_duration_seconds_bucket{method="/csi.v1.Controller/CreateVolume",le="+Inf"} 23
localdir_csi_rpc_duration_seconds_sum{method="/csi.v1.Controller/CreateVolume"} 0.004305040999999999
localdir_csi_rpc_duration_seconds_count{method="/csi.v1.Controller/CreateVolume"} 23
# HELP localdir_csi_rpc_requests_total Total number of RPCs handled, by method and result code.
# TYPE localdir_csi_rpc_requests_total counter
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Controller/CreateSnapshot"} 34
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Controller/CreateVolume"} 23
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Controller/DeleteSnapshot"} 24
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Controller/DeleteVolume"} 36
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Identity/GetPluginInfo"} 3
localdir_csi_rpc_requests_total{code="OK",method="/csi.v1.Identity/Probe"} 2

Every counter above is real — a scrape off a real localdir-csi-controller pod on a kind cluster, after this book's own test traffic (the Chapter 10 snapshot proofs, a handful of CreateVolume calls) had already run against it. CreateVolume alone recorded 23 calls, all code="OK", and its histogram's +Inf bucket — 23 — equals its total count, exactly the cumulative-bucket property explained above. Read the same numbers off your own pod with the three commands above and they'll differ in the counts, not in the shape.

What you should have now

• internal/driver/meta.go's readMetaOrZero, collapsing a read/compare/decide shape that had been hand-written three times — in NodePublishVolume (Chapter 6), CreateVolume (Chapter 9), and CreateSnapshot (Chapter 10) — into one generic helper, all three call sites updated, all 46 pre-existing tests still green with no behavior change
• Confirmation, from a real audit against the CSI spec's own error-code table, that every other already-known simplification (DeleteVolume's missing FAILED_PRECONDITION, ControllerPublishVolume's missing node-not-found NOT_FOUND) is still exactly what Chapter 7 and Chapter 8 already said it was — nothing new to fix, this time
• internal/driver/capabilities.go's supportsVolumeCapability, closing a real gap the error-code audit itself missed: CreateVolume, ControllerPublishVolume, and NodePublishVolume now reject any capability ValidateVolumeCapabilities would already have refused, instead of silently accepting it
• internal/driver/paths.go's safeChildPath, closing a second real gap: every RPC that joins a caller-supplied volume or snapshot ID onto d.dataDir now rejects traversal attempts, embedded separators, and collisions with this driver's own reserved directories, instead of handing an unvalidated opaque string straight to os.MkdirAll or os.RemoveAll
• internal/driver/logging.go and internal/driver/metrics.go: two independent gRPC unary interceptors, each tested by calling it directly with a stub handler, wired into main.go with grpc.ChainUnaryInterceptor — no RPC method anywhere logs or instruments itself
• k8s.io/klog/v2 and github.com/prometheus/client_golang, both real dependencies added with plain go get, the same way this book has added every other module since Chapter 2 — no fakes, no replace directives, nothing this chapter's tests couldn't compile against on a fresh clone
• prometheus.MustRegister wired into internal/driver/metrics.go, and a :9090 HTTP server started alongside the gRPC server in main.go, serving promhttp.Handler() at /metrics — the difference between recording a metric and anyone actually being able to read it
• Real confirmation, off a redeployed pod on the user's own kind cluster via kubectl port-forward and curl, that localdir_csi_rpc_requests_total and localdir_csi_rpc_duration_seconds show up on /metrics exactly as reasoned through above — not just a unit test reading the same counters back via testutil.ToFloat64
• All 58 tests, gofmt, go vet, and go build clean across the whole module — the same bar every chapter since Chapter 3 has held to

Two honest gaps remain. The metrics HTTP server has no TLS and no auth — fine for a kubectl port-forward demo, not fine for a real multi-tenant cluster, where a real deployment would either put it behind a NetworkPolicy restricting who can reach port 9090, or front it with something that authenticates scrapes. And nothing here wires up a Prometheus ServiceMonitor or a Service object for automatic discovery — this chapter proves the endpoint exists and answers correctly; pointing a real Prometheus Operator at it automatically, rather than a manual port-forward, is ordinary Kubernetes plumbing, not something specific to this driver.

Chapter 12 turns to proving all of this automatically: real Go tests against a real kind cluster, codifying the manual kubectl and grpcurl sequences this book has run by hand since Chapter 8 — including a regression test for the real CreateSnapshot idempotency bug Chapter 10 hit live.

Chapter 12: Integration Testing

Every real-cluster proof in this book so far has the same shape: type a kubectl command, read the output, decide whether it matches what the prose predicted. Chapter 8 proved ControllerPublishVolume this way. Chapter 10 proved CreateSnapshot's idempotency this way — and also, on a live cluster, proved a real bug this way, the one where a genuine retry collided with a file os.CopyFS had already written. That bug was caught by a human happening to run the same command twice and noticing the second response wasn't 0 OK. A human doesn't always happen to do that. A test does it every time, on purpose, without needing anyone to remember to check.

This chapter doesn't add anything to the driver. It takes proofs this book has already done by hand — provisioning a volume, mounting it into a pod, snapshotting it, retrying that snapshot — and writes them down as real Go tests that run against a real kind cluster. The payoff is concrete: the exact CreateSnapshot regression from Chapter 10 gets a test that would have caught it the day it happened, instead of relying on whoever's at the keyboard to run the same call twice and read the diff.

client-go, not kubectl

Every object this chapter touches — creating a PersistentVolumeClaim, watching it bind, running a command inside a pod — has a real, first-class API for it in k8s.io/client-go, the same library Kubernetes' own controllers (and kubectl itself, underneath) are built on. Reaching for that library instead of shelling out to kubectl means every check in this chapter reads a real typed Go value (pvc.Status.Phase, not a string parsed out of kubectl get ... -o jsonpath=...), and a failure a step earlier — a client that never connected, a malformed object — surfaces as a real Go error at the exact call that caused it, not as an opaque non-zero exit code from a subprocess three layers removed from the test that's checking it.

One operation in this chapter has no client-go equivalent, and is named honestly rather than faked: checking the kind node's own disk directly, the same docker exec csi-dev-control-plane cat ... proof Chapters 8 and 10 ran by hand. That was never a Kubernetes operation even when kubectl did it — kubectl has no subcommand for it either, which is why those chapters reached for docker exec instead of kubectl exec at exactly that step. client-go talks to the Kubernetes API server; the API server has no concept of "the file sitting on a kind node's host filesystem outside any container." That one check stays os/exec-wrapped docker exec, unavoidably, for the same reason it always was.

Where these tests live, and how they're kept separate

Real-cluster tests need a real cluster — they can't run in the same go test ./... that Chapters 3–11 have relied on for everything else, because that command runs on every commit, on any machine, with no kind cluster guaranteed to exist. Go's answer to "these tests need something extra" is a build tag. test/integration/integration_test.go starts with one:

//go:build integration

package integration

go test ./... (no tags) never even compiles this file — the ordinary test suite Chapters 3–11 built stays exactly as fast and as cluster-independent as it's always been. Running these tests needs an explicit opt-in:

go test -tags=integration ./test/integration/... -v

Add a Makefile target so nobody has to remember that flag, the same way Chapter 9 added sanity-run for csi-sanity:

integration-test:
	@kubectl config current-context 2>/dev/null | grep -qx kind-csi-dev || \
		{ echo "current kubectl context is not kind-csi-dev — run 'make kind-up deploy' or 'kubectl config use-context kind-csi-dev' first" >&2; exit 1; }
	go test -tags=integration ./test/integration/... -v -timeout 5m

The context check above matters for a reason that has nothing to do with correctness of the tests themselves: requireCluster's own context-and-driver checks (below) are deliberately built to t.Skip, not fail, because a stray go test ./... with no cluster around should report "nothing to test here," not a false failure. But that same t.Skip means make integration-test — the thing a reader (or CI) actually runs on purpose, expecting real coverage — would otherwise print ok with every test skipped and zero real assertions run, indistinguishable from a genuinely green suite. The kubectl config current-context check above closes exactly that gap: once someone has explicitly asked for the integration suite to run, "there's no cluster to run it against" is a failure of that request, not a quiet no-op.

client-go is a new real dependency — nothing in internal/driver needed it, so it isn't in go.mod yet. Pin all three of client-go, apimachinery, and api to the same Kubernetes minor version — mixing versions across these three modules is unsupported and will surface as confusing type-mismatch errors, not a version-solver warning:

go get k8s.io/client-go@v0.34.0
go get k8s.io/apimachinery@v0.34.0
go get k8s.io/api@v0.34.0

k8s.io/api/core/v1 shows up directly in this chapter's imports (corev1.PersistentVolumeClaim, corev1.Pod, ...) even though nothing above named it explicitly — go mod tidy after writing the test files below would catch this too, but it's worth adding up front so go build doesn't stall on a missing module mid-chapter.

Setting up the cluster

Every test in this chapter needs the driver actually running on a real kind cluster first — none of them stand up a cluster themselves, the same way Chapters 8 and 10's grpcurl proofs never did either. This is the same sequence those chapters walked through by hand, run once before touching any test file. From the project root:

make kind-up
make deploy

Creating cluster "csi-dev" ...
 ✓ Ensuring node image (kindest/node:v1.36.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-csi-dev"
...
kubectl apply -f deploy/
serviceaccount/localdir-csi-controller created
clusterrole.rbac.authorization.k8s.io/localdir-csi-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/localdir-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/localdir-csi-snapshotter created
clusterrolebinding.rbac.authorization.k8s.io/localdir-csi-snapshotter created
deployment.apps/localdir-csi-controller created
csidriver.storage.k8s.io/localdir.csi.example.com created
daemonset.apps/localdir-csi-node created
storageclass.storage.k8s.io/localdir-csi created
error: resource mapping not found for name: "localdir-csi-snapshot" ... no matches for kind "VolumeSnapshotClass"
ensure CRDs are installed first

That last error is expected, and safe to ignore for this chapter: deploy/volumesnapshotclass.yaml (Chapter 10) needs the external-snapshotter CRDs installed cluster-wide first, and none of this chapter's tests go through a VolumeSnapshot object — TestCreateSnapshot_IdempotentRetry calls CreateSnapshot directly by grpcurl, the same as Chapter 10's own grpcurl proof did before its VolumeSnapshot section. Everything else applied cleanly. Confirm both pods came up:

kubectl get pods

NAME                                       READY   STATUS    RESTARTS   AGE
localdir-csi-controller-7f56558f88-tqxhk   3/3     Running   0          15s
localdir-csi-node-9s6bz                    2/2     Running   0          15s

One more manual check worth running once, by hand, before trusting the suite to run it automatically: confirm grpcurl itself actually works against this pod, the same kubectl cp step Chapters 8 and 10 ran by hand. (TestCreateSnapshot_IdempotentRetry below doesn't depend on this having been run — it uploads its own copy of grpcurl every time, so a pod restart between now and then can't silently break it — but seeing a real response here first is worth the thirty seconds.)

CTRL_POD=$(kubectl get pods -l app=localdir-csi-controller -o jsonpath='{.items[0].metadata.name}')
kubectl cp ./grpcurl-linux $CTRL_POD:/tmp/grpcurl -c localdir-csi
kubectl exec $CTRL_POD -c localdir-csi -- chmod +x /tmp/grpcurl
kubectl exec $CTRL_POD -c localdir-csi -- /tmp/grpcurl -plaintext unix:///csi/csi.sock csi.v1.Identity/GetPluginInfo

{
  "name": "localdir.csi.example.com",
  "vendorVersion": "0.1.0"
}

A real response from a real running driver — the cluster is ready for every test below. kind clusters survive between go test runs (they aren't torn down by anything in this chapter), so this setup normally only needs doing once per session; make kind-down tears it back down when you're done for good.

Connecting: one client, shared by every test

Every test in this chapter needs the same thing first — a *kubernetes.Clientset built from whatever kubeconfig kubectl itself would use, so "the cluster this test talks to" is always the same cluster kubectl would have talked to by hand. client-go's clientcmd package resolves that the same way kubectl does: KUBECONFIG, then ~/.kube/config. Rather than hand-write that resolution in every test, one helper, in test/integration/cluster_test.go, that everything else in this chapter goes through — the same "one seam" shape Chapter 11's interceptors already taught.

"Whatever kubeconfig kubectl would use" is convenient, but it cuts both ways: this suite creates Pods and PVCs, and it deletes them in cleanup. If your current context happens to point at some other cluster — one you were debugging five minutes ago, say — "convenient" becomes "this test just created and deleted real objects somewhere it had no business touching." requireCluster doesn't just check that a cluster is reachable; it checks that the reachable cluster is this tutorial's disposable kind cluster, by name and by the presence of the driver these tests are actually exercising, before returning anything a test could use to mutate it:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
	"time"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
)

// wantContext is the only kubeconfig context these tests will run
// against. `kind create cluster --name csi-dev` (Chapter 2) always
// names its context kind-<cluster-name>, so this matches this book's
// own setup exactly — change it if you named your cluster something
// else.
const wantContext = "kind-csi-dev"

// wantCSIDriverName is the CSIDriver object Chapter 4 registers. Its
// presence is this package's proof that the reachable cluster is
// running this tutorial's driver, not just any kind cluster that
// happens to be named the same thing.
const wantCSIDriverName = "localdir.csi.example.com"

// requireCluster builds a Clientset from whatever kubeconfig kubectl
// itself would use (KUBECONFIG, then ~/.kube/config), and refuses to
// return one at all unless that kubeconfig's current context is
// wantContext and the cluster it points at is actually running
// wantCSIDriverName. Every test in this package calls this first —
// and skips, rather than fails confusingly three calls later, or
// silently mutates the wrong cluster — if either check fails.
func requireCluster(t *testing.T) (*kubernetes.Clientset, *rest.Config) {
	t.Helper()

	loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
	rawConfig, err := loadingRules.Load()
	if err != nil {
		t.Skipf("no usable kubeconfig (%v) — run against a real kind cluster with the driver deployed", err)
	}
	if rawConfig.CurrentContext != wantContext {
		t.Skipf("current kubeconfig context is %q, want %q — these tests create and delete real objects and must not run against the wrong cluster; run `kubectl config use-context %s` first",
			rawConfig.CurrentContext, wantContext, wantContext)
	}

	restConfig, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		loadingRules, &clientcmd.ConfigOverrides{},
	).ClientConfig()
	if err != nil {
		t.Skipf("no usable kubeconfig (%v) — run against a real kind cluster with the driver deployed", err)
	}

	clientset, err := kubernetes.NewForConfig(restConfig)
	if err != nil {
		t.Fatalf("kubernetes.NewForConfig: %v", err)
	}

	if _, err := clientset.Discovery().ServerVersion(); err != nil {
		t.Skipf("cluster unreachable (%v) — run against a real kind cluster with the driver deployed", err)
	}

	if _, err := clientset.StorageV1().CSIDrivers().Get(context.Background(), wantCSIDriverName, metav1.GetOptions{}); err != nil {
		t.Skipf("context %q is reachable, but CSIDriver %q isn't registered on it (%v) — deploy this tutorial's driver first with `make deploy`",
			wantContext, wantCSIDriverName, err)
	}

	return clientset, restConfig
}

// uniqueName appends a nanosecond timestamp to base, so re-running this
// suite immediately after a previous run — or a previous run's cleanup
// only partially finishing, PVC finalizers included — never collides
// with an object the previous run created. Every test below calls this
// once per object it creates, rather than hardcoding a fixed name.
func uniqueName(base string) string {
	return fmt.Sprintf("%s-%d", base, time.Now().UnixNano())
}

Both new checks fail closed with t.Skip, not t.Fatal — the same "there's nothing wrong with your code, there's nothing here to test yet" signal the reachability check already used. A context-name mismatch or a missing CSIDriver isn't this suite's problem to fix; it's a sign the reader hasn't pointed kubectl at the right place yet, and the right response is to say so and stop, not to touch anything.

restConfig gets returned alongside clientset because building an authenticated connection into a running pod (needed further down, for the grpcurl and file-write proofs) reuses the exact same credentials clientcmd already resolved — no second round of "where's the kubeconfig" logic anywhere else in this package.

Waiting for a status to change

Nothing in the Kubernetes API blocks until a PersistentVolumeClaim says Bound — a Get call right after Create almost always still says Pending. k8s.io/apimachinery's own wait package is the real library idiom for exactly this shape of problem, the same package Kubernetes' own controllers use to poll for a condition. Add it to test/integration/cluster_test.go, alongside requireCluster — they're both small, cluster-connection-adjacent helpers every other test in this package calls. Add "context", "time", and "k8s.io/apimachinery/pkg/util/wait" to that file's existing import block (alongside "testing" and the three k8s.io/client-go imports requireCluster already needed), then add the function itself:

// waitFor polls condition every 500ms until it returns true, or fails
// the test once timeout has passed. condition returning a non-nil error
// stops the poll immediately and fails the test with that error, the
// same way a real setup problem (not just "not ready yet") should.
func waitFor(t *testing.T, timeout time.Duration, what string, condition wait.ConditionWithContextFunc) {
	t.Helper()
	if err := wait.PollUntilContextTimeout(context.Background(), 500*time.Millisecond, timeout, true, condition); err != nil {
		t.Fatalf("timed out waiting for %s: %v", what, err)
	}
}

condition's real type, wait.ConditionWithContextFunc, is func(ctx context.Context) (done bool, err error) — every call site below hands waitFor a closure matching that shape, reading whatever object it's waiting on fresh off the API server each time.

Running a command inside a pod

kubectl exec has a client-go equivalent too, though it's the one piece in this chapter with real ceremony behind it: exec runs over a separate, upgraded HTTP connection (SPDY), not an ordinary REST call, because the pod's stdout is a live stream, not a JSON response body. One helper hides that ceremony from every test that needs it, in test/integration/exec_test.go:

//go:build integration

package integration

import (
	"bytes"
	"context"
	"os"
	"testing"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/kubernetes/scheme"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/remotecommand"
)

// execInPod runs command inside container of pod podName, in namespace
// ns, and returns its combined stdout+stderr. It fails the test on a
// transport error (couldn't even reach the pod) or a non-zero exit
// inside the container, both surfaced as a real Go error from
// remotecommand rather than a parsed shell exit code. ctx is taken as
// an explicit parameter rather than reaching for t.Context() inside
// the function — more on why right after this listing.
func execInPod(ctx context.Context, t *testing.T, restConfig *rest.Config, clientset *kubernetes.Clientset, ns, podName, container string, command []string) string {
	t.Helper()

	req := clientset.CoreV1().RESTClient().Post().
		Resource("pods").
		Namespace(ns).
		Name(podName).
		SubResource("exec")
	req.VersionedParams(&corev1.PodExecOptions{
		Container: container,
		Command:   command,
		Stdout:    true,
		Stderr:    true,
	}, scheme.ParameterCodec)

	executor, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
	if err != nil {
		t.Fatalf("building exec request for pod %q: %v", podName, err)
	}

	var stdout, stderr bytes.Buffer
	err = executor.StreamWithContext(ctx, remotecommand.StreamOptions{
		Stdout: &stdout,
		Stderr: &stderr,
	})
	combined := stdout.String() + stderr.String()
	if err != nil {
		t.Fatalf("exec %v in pod %q: %v\n%s", command, podName, err, combined)
	}
	return combined
}

// uploadGrpcurl copies the local grpcurl-linux binary into podName's
// container at /tmp/grpcurl, over the same exec stream execInPod uses
// — just with Stdin attached instead of Stdout/Stderr captured, the
// same way `kubectl cp` itself streams a file through exec underneath.
// Any test that needs grpcurl calls this first, every run, rather than
// assuming a copy from a previous session survived — a controller pod
// restart (a new rollout, a node reboot) wipes /tmp along with it.
func uploadGrpcurl(ctx context.Context, t *testing.T, restConfig *rest.Config, clientset *kubernetes.Clientset, ns, podName, container string) {
	t.Helper()

	f, err := os.Open("grpcurl-linux")
	if err != nil {
		t.Fatalf("opening local grpcurl-linux binary (build it per Chapter 8's grpcurl setup): %v", err)
	}
	defer f.Close()

	req := clientset.CoreV1().RESTClient().Post().
		Resource("pods").
		Namespace(ns).
		Name(podName).
		SubResource("exec")
	req.VersionedParams(&corev1.PodExecOptions{
		Container: container,
		Command:   []string{"sh", "-c", "cat > /tmp/grpcurl && chmod +x /tmp/grpcurl"},
		Stdin:     true,
		Stdout:    true,
		Stderr:    true,
	}, scheme.ParameterCodec)

	executor, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
	if err != nil {
		t.Fatalf("building exec request for pod %q: %v", podName, err)
	}

	var stdout, stderr bytes.Buffer
	if err := executor.StreamWithContext(ctx, remotecommand.StreamOptions{
		Stdin:  f,
		Stdout: &stdout,
		Stderr: &stderr,
	}); err != nil {
		t.Fatalf("uploading grpcurl to pod %q: %v\n%s", podName, err, stdout.String()+stderr.String())
	}
}

os.Open("grpcurl-linux") resolves relative to go test's own working directory, which is the package directory — so this expects the binary at test/integration/grpcurl-linux, not the project root. Copy or symlink it there once; go test doesn't change directories between runs, so this, unlike the pod-side copy, only needs doing once per checkout.

scheme.ParameterCodec (from k8s.io/client-go/kubernetes/scheme) is what turns the typed PodExecOptions struct — container name, the command to run, which streams to attach — into the URL query parameters the API server's exec subresource actually expects; it's the same encoding kubectl exec relies on internally, just reached directly instead of through a subprocess.

Two things about this listing are corrections, not stylistic choices, and both were found the same way every real bug in this book has been found: by running the thing against a real cluster and watching it misbehave.

The first: ctx is a parameter here, not t.Context() called directly inside the function, because execInPod needs to work correctly from inside a t.Cleanup closure too — Chapter's own TestCreateSnapshot_IdempotentRetry below calls grpcurl (which wraps execInPod) from both the test body and its cleanup, to delete what it created. t.Context() is documented to be "canceled just before Cleanup-registered functions are called" — always, every run — so a version of execInPod that called t.Context() internally would work fine everywhere except cleanup, where every call would fail with operation was canceled. Confirmed live: an earlier version of this exact helper did exactly that, and its cleanup's DeleteSnapshot/DeleteVolume calls failed with precisely that error the first time this chapter's snapshot test ran for real. Passing ctx in means callers decide: t.Context() from the test body, context.Background() from inside t.Cleanup — the same fix TestCreateVolume_ViaPVC's own cleanup already needed, below.

The second is subtler, and was the real cause of a much stranger symptom: an early version of this helper pointed both Stdout and Stderr at the same bytes.Buffer. remotecommand streams stdout and stderr back on separate channels, read by separate goroutines — and bytes.Buffer is not safe for concurrent writes. Running TestCreateSnapshot_IdempotentRetry for real against this book's kind cluster reproduced this exactly: the two CreateSnapshot calls occasionally came back with one of them holding a real, correct response and the other holding an empty string, with no error from either call — a plain data race, not a driver bug, not a network flake, confirmed by re-running the same two grpcurl calls by hand with kubectl exec (no client-go involved) and seeing identical, correct output both times. go test -race on the fixed version — two separate buffers, concatenated only after StreamWithContext returns, once both goroutines are known to be done writing — ran clean across several real, repeated runs against the cluster. The lesson generalizes past this one helper: any time two ends of a concurrent stream get pointed at the same non-thread-safe destination, "sometimes empty, never errors" is exactly the shape of bug to expect.

Two object builders, shared instead of duplicated

Two tests below both need a PersistentVolumeClaim shaped exactly like Chapter 8's demo-pvc.yaml, and one needs a Pod shaped exactly like demo-pod.yaml. Writing that struct literal twice would be exactly the kind of repetition Chapter 6 already taught this book to notice and factor out — so both live once, in test/integration/objects_test.go:

//go:build integration

package integration

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/api/resource"
)

// demoPVC builds a PersistentVolumeClaim matching Chapter 8's
// demo-pvc.yaml exactly — same storage class, access mode, and size.
func demoPVC(name string) *corev1.PersistentVolumeClaim {
	storageClassName := "localdir-csi"
	return &corev1.PersistentVolumeClaim{
		ObjectMeta: metav1.ObjectMeta{Name: name},
		Spec: corev1.PersistentVolumeClaimSpec{
			StorageClassName: &storageClassName,
			AccessModes:      []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
			Resources: corev1.VolumeResourceRequirements{
				Requests: corev1.ResourceList{
					corev1.ResourceStorage: resource.MustParse("1Gi"),
				},
			},
		},
	}
}

// demoPod builds a Pod matching Chapter 8's demo-pod.yaml exactly —
// same image, same mount path, referencing whichever PVC name is passed
// in so TestNodePublishVolume_ViaPod and any future test can reuse it
// without hardcoding "demo-claim" a second time.
func demoPod(name, pvcName string) *corev1.Pod {
	return &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: name},
		Spec: corev1.PodSpec{
			Containers: []corev1.Container{{
				Name:    "app",
				Image:   "busybox",
				Command: []string{"sleep", "3600"},
				VolumeMounts: []corev1.VolumeMount{{
					Name:      "data",
					MountPath: "/data",
				}},
			}},
			Volumes: []corev1.Volume{{
				Name: "data",
				VolumeSource: corev1.VolumeSource{
					PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{
						ClaimName: pvcName,
					},
				},
			}},
		},
	}
}

Provisioning: CreateVolume, triggered by a PersistentVolumeClaim

Chapter 8's proof: create the claim, watch it go Pending → Bound, confirm a PersistentVolume now exists. As a test, in test/integration/provisioning_test.go:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func TestCreateVolume_ViaPVC(t *testing.T) {
	clientset, _ := requireCluster(t)
	ctx := t.Context()
	pvcs := clientset.CoreV1().PersistentVolumeClaims("default")
	claimName := uniqueName("it-demo-claim")

	if _, err := pvcs.Create(ctx, demoPVC(claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating PVC: %v", err)
	}
	t.Cleanup(func() {
		bg := context.Background()
		if err := pvcs.Delete(bg, claimName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting PVC %q: %v", claimName, err)
			return
		}
		waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to be gone", claimName), func(ctx context.Context) (bool, error) {
			_, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
			if apierrors.IsNotFound(err) {
				return true, nil
			}
			return false, err
		})
	})

	var bound *corev1.PersistentVolumeClaim
	waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to bind", claimName), func(ctx context.Context) (bool, error) {
		pvc, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
		if err != nil {
			return false, err
		}
		if pvc.Status.Phase == corev1.ClaimBound {
			bound = pvc
			return true, nil
		}
		return false, nil
	})

	if bound.Spec.VolumeName == "" {
		t.Fatal("PVC bound, but Spec.VolumeName is empty")
	}

	pv, err := clientset.CoreV1().PersistentVolumes().Get(ctx, bound.Spec.VolumeName, metav1.GetOptions{})
	if err != nil {
		t.Fatalf("getting PersistentVolume %q: %v", bound.Spec.VolumeName, err)
	}
	if pv.Status.Phase != corev1.VolumeBound {
		t.Errorf("PersistentVolume %q status = %q, want %q", pv.Name, pv.Status.Phase, corev1.VolumeBound)
	}
}

t.Cleanup runs even if an assertion above it fails — the same reason Chapters 6–10's own newTestDriverInTempDir helper uses it — so a failed run doesn't leave the claim (and the real directory CreateVolume made for it) behind for the next run to trip over. uniqueName means a leftover claim from an earlier failed run can never collide with this run's name either way, but the cleanup still checks the Delete call's own error with t.Errorf — a silently swallowed error here would hide a real problem (RBAC, a finalizer stuck forever) behind a green test — and then waitFors the claim actually disappearing rather than firing the delete and assuming it worked; PersistentVolumeClaim carries the pvc-protection finalizer, so Get can still return the object, Terminating, for a moment after Delete returns.

The cleanup deliberately uses context.Background(), not ctx, and this one is worth being exact about, because the bug it avoids is real and easy to reintroduce by reaching for the context already in scope: Go's own testing package documents t.Context() as "canceled just before Cleanup-registered functions are called" — not "eventually," not "maybe," but always, on every run, pass or fail. A Delete call made with ctx inside t.Cleanup is handed an already-cancelled context before it even reaches the API server, and client-go returns immediately with a context-cancellation error — which an unchecked cleanup wouldn't even notice, so the delete would silently never happen. Confirmed live, against a real cluster, while drafting this chapter: a first version using ctx in t.Cleanup left the claim sitting there, Bound, after a passing test run. context.Background() in cleanup, always — the same lesson TestNodePublishVolume_ViaPod below already applies.

Run it now, against the cluster from the setup section above:

go test -tags=integration ./test/integration/... -run TestCreateVolume_ViaPVC -v

=== RUN   TestCreateVolume_ViaPVC
--- PASS: TestCreateVolume_ViaPVC (0.52s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	0.530s

Real 0 OK, against a real cluster — and worth checking the thing the test itself can't check about its own cleanup:

kubectl get pvc

No resources found in default namespace.

Gone, confirming the context.Background() fix actually took — exactly the same proof-that-cleanup-really-cleaned-up instinct every docker exec ... ls check since Chapter 8 has used, just aimed at a test's own teardown this time instead of the driver's.

Mounting: NodePublishVolume, triggered by a Pod

Chapter 8's other half: a pod that references the claim gets kubelet to call NodePublishVolume with no grpcurl involved, and a file written inside the pod is provably the same file sitting in the node's own directory. In test/integration/mount_test.go:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"os/exec"
	"strings"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

const kindNodeContainer = "csi-dev-control-plane"

func TestNodePublishVolume_ViaPod(t *testing.T) {
	clientset, restConfig := requireCluster(t)
	ctx := t.Context()
	pvcs := clientset.CoreV1().PersistentVolumeClaims("default")
	pods := clientset.CoreV1().Pods("default")
	claimName := uniqueName("it-mount-claim")
	podName := uniqueName("it-mount-pod")

	if _, err := pvcs.Create(ctx, demoPVC(claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating PVC: %v", err)
	}
	if _, err := pods.Create(ctx, demoPod(podName, claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating Pod: %v", err)
	}
	t.Cleanup(func() {
		bg := context.Background()
		if err := pods.Delete(bg, podName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting pod %q: %v", podName, err)
		} else {
			waitFor(t, 30*time.Second, fmt.Sprintf("pod %s to be gone", podName), func(ctx context.Context) (bool, error) {
				_, err := pods.Get(ctx, podName, metav1.GetOptions{})
				if apierrors.IsNotFound(err) {
					return true, nil
				}
				return false, err
			})
		}
		if err := pvcs.Delete(bg, claimName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting PVC %q: %v", claimName, err)
			return
		}
		waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to be gone", claimName), func(ctx context.Context) (bool, error) {
			_, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
			if apierrors.IsNotFound(err) {
				return true, nil
			}
			return false, err
		})
	})

	waitFor(t, 60*time.Second, fmt.Sprintf("pod %s to become Running", podName), func(ctx context.Context) (bool, error) {
		pod, err := pods.Get(ctx, podName, metav1.GetOptions{})
		if err != nil {
			return false, err
		}
		return pod.Status.Phase == corev1.PodRunning, nil
	})

	execInPod(ctx, t, restConfig, clientset, "default", podName, "app",
		[]string{"sh", "-c", "echo integration-test > /data/hello.txt"})

	fromPod := execInPod(ctx, t, restConfig, clientset, "default", podName, "app",
		[]string{"cat", "/data/hello.txt"})
	if strings.TrimSpace(fromPod) != "integration-test" {
		t.Fatalf("cat inside pod = %q, want %q", fromPod, "integration-test")
	}

	pvc, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
	if err != nil {
		t.Fatalf("getting PVC: %v", err)
	}
	nodePath := "/var/lib/localdir-csi/data/" + pvc.Spec.VolumeName + "/hello.txt"
	out, err := exec.Command("docker", "exec", kindNodeContainer, "cat", nodePath).Output()
	if err != nil {
		t.Fatalf("docker exec cat %s: %v", nodePath, err)
	}
	if fromNode := strings.TrimSpace(string(out)); fromNode != "integration-test" {
		t.Errorf("cat on node disk = %q, want %q", fromNode, "integration-test")
	}
}

Same proof-from-both-sides technique every hand-run chapter since Chapter 6 has used, just run by go test instead of a person: the pod sees the file it wrote, and the node's own disk — reached through a completely separate path, docker exec into the kind container rather than a pod exec stream — sees the exact same content. If the driver only pretended to bind-mount (say, it silently wrote into a scratch directory pods never actually see), this test would still pass its first assertion and fail its second — which is exactly why the second one exists.

strings.TrimSpace on both sides of that comparison is not decoration — cat's output over a real exec stream carries the trailing newline echo wrote, same as cat always would from a real terminal. Compare fromPod (or fromNode) without trimming it first and this test fails on a difference that was never actually a difference in the data, only in whitespace neither side cares about — a real, easy mistake, not a hypothetical one.

Run it now:

go test -tags=integration ./test/integration/... -run TestNodePublishVolume_ViaPod -v

=== RUN   TestNodePublishVolume_ViaPod
--- PASS: TestNodePublishVolume_ViaPod (2.14s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	2.143s

One more real wrinkle worth naming, hit while confirming this test back-to-back with itself in a tight loop: a PersistentVolumeClaim doesn't finish deleting the instant Delete returns — a pvc-protection finalizer holds it in Terminating until the underlying volume is actually gone, which for this driver means DeleteVolume has to run first. Reusing a fixed claim name across runs, with a cleanup that fires Delete and moves on without checking whether the object is actually gone, can transiently hit object is being deleted: ... already exists on the next run's Create if the previous run's Terminating claim hasn't finished clearing yet. That's exactly why both this test's and TestCreateVolume_ViaPVC's cleanups call waitFor on the delete, not just the create — a cleanup that returns before the object is actually gone isn't really "cleaned up," it's "asked to clean up and hoped." Combined with uniqueName, a Terminating leftover from a genuinely interrupted run (a killed test process, say) still can't collide with anything a later run creates, even if it clears too slowly to matter to that later run at all.

The regression test: CreateSnapshot, called twice

This is the test this chapter exists to write. Chapter 10's live incident, replayed exactly, as TestCreateSnapshot_IdempotentRetry in test/integration/snapshot_test.go. Calling the driver directly by grpcurl — run inside the controller pod via the same execInPod helper above, rather than through a VolumeSnapshot object — keeps this test aimed precisely at the bug — a VolumeSnapshotContent retry loop would eventually paper over a slow second call in ways that would make a real regression harder to catch, not easier. Finding the controller pod itself is one more client-go list call, by the same label kubectl logs -l app=localdir-csi-controller has used by hand since Chapter 7:

//go:build integration

package integration

import (
	"context"
	"testing"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
)

func controllerPodName(t *testing.T, clientset *kubernetes.Clientset) string {
	t.Helper()
	pods, err := clientset.CoreV1().Pods("default").List(t.Context(), metav1.ListOptions{
		LabelSelector: "app=localdir-csi-controller",
	})
	if err != nil {
		t.Fatalf("listing controller pods: %v", err)
	}
	if len(pods.Items) == 0 {
		t.Fatal("no localdir-csi-controller pod found — is the driver deployed?")
	}
	return pods.Items[0].Name
}

With that in place, the regression test itself:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
)

func TestCreateSnapshot_IdempotentRetry(t *testing.T) {
	clientset, restConfig := requireCluster(t)
	ctx := t.Context()
	podName := controllerPodName(t, clientset)
	uploadGrpcurl(ctx, t, restConfig, clientset, "default", podName, "localdir-csi")

	volumeID := uniqueName("it-snap-source")
	snapshotID := uniqueName("it-snap-demo")
	grpcurl := func(ctx context.Context, data, method string) string {
		return execInPod(ctx, t, restConfig, clientset, "default", podName, "localdir-csi", []string{
			"/tmp/grpcurl", "-plaintext", "-d", data,
			"unix:///csi/csi.sock", "csi.v1.Controller/" + method,
		})
	}

	t.Cleanup(func() {
		bg := context.Background()
		grpcurl(bg, fmt.Sprintf(`{"snapshot_id": %q}`, snapshotID), "DeleteSnapshot")
		grpcurl(bg, fmt.Sprintf(`{"volume_id": %q}`, volumeID), "DeleteVolume")
	})

	grpcurl(ctx, fmt.Sprintf(`{
		"name": %q,
		"capacity_range": {"required_bytes": 1048576},
		"volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
	}`, volumeID), "CreateVolume")

	execInPod(ctx, t, restConfig, clientset, "default", podName, "localdir-csi",
		[]string{"sh", "-c", fmt.Sprintf("echo real data > /data/%s/hello.txt", volumeID)})

	first := grpcurl(ctx, fmt.Sprintf(`{
		"source_volume_id": %q,
		"name": %q
	}`, volumeID, snapshotID), "CreateSnapshot")

	second := grpcurl(ctx, fmt.Sprintf(`{
		"source_volume_id": %q,
		"name": %q
	}`, volumeID, snapshotID), "CreateSnapshot")

	if first != second {
		t.Errorf("CreateSnapshot retry returned a different response:\nfirst:  %s\nsecond: %s", first, second)
	}
}

uploadGrpcurl replaces the manual kubectl cp step from the cluster setup section above — that step is still worth knowing (it's exactly what uploadGrpcurl automates), but this test no longer depends on it having been run, or on its result having survived whatever's happened to the controller pod since.

grpcurl itself takes ctx as a parameter, for the exact same reason execInPod does — this closure is called from both the test body (where ctx is t.Context()) and from t.Cleanup (where it has to be context.Background() instead). A closure that captured a single fixed ctx from its enclosing scope, rather than taking one as a parameter, would silently pick whichever context was in scope when the closure was defined — the test body's, canceled by the time cleanup runs — the same bug in a slightly more hidden shape.

first != second is the entire regression check, and it's worth being precise about why that one comparison is enough. grpcurl's JSON output includes creationTime, down to the nanosecond. Chapter 10's real incident didn't fail this way — the pre-fix bug returned a real gRPC error (Internal: ... file exists) on the second call, which execInPod's t.Fatalf on a non-zero exit would already catch, before the string comparison ever runs. The string comparison catches the other failure mode a fix like this can quietly introduce: a second call that "succeeds" but doesn't actually short-circuit — say, one that recomputes sizeBytes from a directory listing instead of returning the recorded metadata. That would return 0 OK both times, pass a check that only asserted "no error," and still be wrong. Comparing the full response, creationTime included, is what makes this test check the same snapshot, not just no error — the exact distinction Chapter 10's real live confirmation drew by hand when it checked that the retry's creationTime matched the original down to the nanosecond.

Run it now, the same way as the two tests above:

go test -tags=integration ./test/integration/... -run TestCreateSnapshot_IdempotentRetry -v

=== RUN   TestCreateSnapshot_IdempotentRetry
--- PASS: TestCreateSnapshot_IdempotentRetry (0.24s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	0.252s

Real 0 OK twice, real identical creationTime, confirmed by go test, on the exact bug this chapter set out to catch — worth re-running a few times in a row rather than trusting one green result, precisely because this section's own two fixes above were both found by a previously passing-looking run turning out to be unreliable. Five repeated runs, with go test -race (to keep the buffer fix honest) and a manual cleanup between each:

go test -tags=integration -race ./test/integration/... -run TestCreateSnapshot_IdempotentRetry -v -count=1

--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.30s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.29s)

Five for five, no race detected — the shared-buffer bug is really gone, not just quiet this particular run.

Running the whole suite together

Once all three tests exist, make integration-test runs them in one pass — the same Makefile target added earlier in this chapter:

make integration-test

go test -tags=integration ./test/integration/... -v -timeout 5m
=== RUN   TestNodePublishVolume_ViaPod
--- PASS: TestNodePublishVolume_ViaPod (2.62s)
=== RUN   TestCreateVolume_ViaPVC
--- PASS: TestCreateVolume_ViaPVC (0.51s)
=== RUN   TestCreateSnapshot_IdempotentRetry
--- PASS: TestCreateSnapshot_IdempotentRetry (0.24s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	3.384s

go test runs a package's tests in the order they appear in each file, sorted by filename — TestNodePublishVolume_ViaPod (mount_test.go) before TestCreateVolume_ViaPVC (provisioning_test.go), alphabetically. All three real, all three green, in one pass, against one real kind cluster.

Three tests, each skipping cleanly if run directly with go test and no matching cluster around, but failing loudly if run via make integration-test with the wrong context or no cluster at all: TestCreateVolume_ViaPVC, TestNodePublishVolume_ViaPod, and TestCreateSnapshot_IdempotentRetry — the last one doing, on every run, exactly what a human happened to do once by accident back in Chapter 10.

What you should have now

• test/integration/, a //go:build integration-tagged package, invisible to the ordinary go test ./... every earlier chapter's suite runs
• requireCluster, waitFor, and execInPod: three small helpers everything else in the package builds on, the same "one seam" shape as Chapter 11's interceptors — plus demoPVC/demoPod, two object builders shared by every test that needs one, instead of the same struct literal written out twice
• k8s.io/client-go and k8s.io/apimachinery as new real dependencies — the first time this driver's own go.mod has needed a Kubernetes client library, since every earlier chapter's Kubernetes interaction was either a sidecar's job or typed by hand
• TestCreateVolume_ViaPVC: Chapter 8's PersistentVolumeClaim provisioning proof, automated
• TestNodePublishVolume_ViaPod: Chapter 8's pod-mount proof, automated, keeping the same proof-from-both-sides technique — a client-go exec stream inside the pod, docker exec on the node's own disk (the one operation with no client-go equivalent, named honestly rather than faked)
• TestCreateSnapshot_IdempotentRetry: a real regression test for the exact bug Chapter 10 hit live, comparing the full response of two identical CreateSnapshot calls rather than just checking for an error, so a future regression that "succeeds twice, quietly wrong" gets caught too, not just one that errors outright
• An integration-test Makefile target, the same convenience Chapter 9 added for csi-sanity
• Two real bugs in the test helpers themselves, found only by actually running this chapter's tests against a real cluster repeatedly, not by reading the code: execInPod (and, following the same pattern, TestCreateSnapshot_IdempotentRetry's grpcurl closure) taking ctx as an explicit parameter instead of reaching for t.Context() internally, because t.Context() is canceled before any t.Cleanup runs; and execInPod using two separate buffers for stdout and stderr instead of one shared bytes.Buffer, because two goroutines writing to the same non-thread-safe buffer is a real data race, confirmed with go test -race clean across five repeated runs once fixed. Neither bug was hypothetical — both reproduced live, against this book's own kind cluster, before being fixed
• requireCluster rejecting any kubeconfig context other than kind-csi-dev, and any cluster missing this tutorial's own CSIDriver, before any test can create or mutate anything — the suite no longer trusts "some cluster is reachable" to mean "this is the disposable one"
• uniqueName, used for every PVC, Pod, and CSI volume/snapshot name this chapter creates, so re-running the suite immediately — or after a previous run's cleanup only partially finished — can't collide with a leftover from before
• Every cleanup now checks its own Delete call's error with t.Errorf and waitFors the object actually disappearing, instead of firing a delete and assuming it worked — pvc-protection's finalizer means "deleted" and "gone" aren't the same moment
• uploadGrpcurl, replacing a manually-copied /tmp/grpcurl that a controller pod restart would silently wipe — the snapshot regression test now provisions its own copy of the binary it depends on, every run
• make integration-test failing outright, with a clear message, when the current kubectl context isn't kind-csi-dev — the tests themselves still skip cleanly for an ad hoc go test run with no cluster around, but the one command a reader (or CI) actually invokes on purpose no longer reports ok having silently run zero real assertions

Two honest gaps remain. These tests aren't wired into any CI — they need a real kind cluster with the driver already deployed, and this book has never set up a CI pipeline to provide one; running them today means running them by hand, the same way every other real-cluster proof in this book has been run, just with go test instead of a person reading kubectl output line by line. And coverage here is deliberately narrow — three tests, not a full conformance suite (that's what Chapter 9's csi-sanity already is) — chosen specifically to cover the two flows this book has proven by hand the most (provisioning and mounting) plus the one real bug this book has actually hit live. A fuller integration suite covering every RPC this driver has is possible with exactly these same helpers; it just isn't written yet.