Chapter 8: Attaching Volumes

Chapter 7 gave the Controller service a real CreateVolume and DeleteVolume, proved with grpcurl typed by hand against a real pod. Two things are still missing before this driver behaves the way a real Kubernetes user would actually experience it. First, there's a whole category of the CSI spec this book hasn't touched yet — "attaching" a volume — and it's worth understanding what that means and whether localdir-csi needs it, rather than skipping it silently. Second, and bigger: nothing so far has ever been triggered by Kubernetes itself. Every RPC call in this book has been typed into grpcurl by a person. This chapter closes both gaps.

What "attaching" a volume actually means

Picture a USB drive. Right now it's sitting on your desk, unplugged.

The drive already has data on it. It already has a filesystem. It already has a fixed capacity. None of that depends on being plugged in.

But your laptop can't see any of it yet. It can't mount the drive. It can't read a single file off it. Not because the drive is broken — just because the two aren't connected yet.

Now plug it in. Nothing on the drive changes. No new files appear. The data was always there. What changes is simpler than that: your laptop can suddenly see the drive. That's the whole effect of plugging it in — reachability, not creation.

Unplug it again. Carry it to a different laptop. Plug it in there instead. That laptop can see it now. Usually only one laptop at a time, though — plugging a drive into a second machine while it's still in the first one doesn't work the way you'd want it to.

That plug-in step — making something that already exists reachable from one specific machine — is exactly what ControllerPublishVolume does in CSI. Say it again, plainly: publishing a volume doesn't create it. Publishing a volume just makes it reachable from one node.

Here's where that matters for real. Take an AWS EBS volume. It lives on Amazon's own storage infrastructure, not on any one of your nodes. Before a pod can mount it, someone has to tell AWS: "attach volume vol-123 to instance i-0abc." That one API call is the cloud version of plugging in the USB drive. Skip it, and there's nothing for the node to mount — the same way an unplugged USB drive has nothing for your laptop to mount.

Now back to localdir-csi. Its volumes were never unplugged from anywhere. There was never a separate drive to begin with.

CreateVolume just makes a directory. That directory lives at /var/lib/localdir-csi/data/<volume-id>, right there on the node's own local disk. Chapter 7 already explained why that exact path has to match on every node in this cluster — same reasoning, same path, still true here.

There's no second piece of storage sitting off to the side, waiting to be attached. The "drive," if you want to keep calling it that, was built into the machine from day one. It's like your laptop's own internal disk: you don't plug that in before saving a file to it. It's just already there.

So, one more time, plainly: this driver has nothing to attach, because it never had anything separate to begin with.

That still leaves a real question for this chapter, though. CSI's spec defines ControllerPublishVolume and ControllerUnpublishVolume whether or not a given driver's storage needs them — the two methods exist in the interface either way. So the question isn't "can we skip writing these methods." It's "should this driver use them the same way a driver with a real attach step would." And the honest answer is no.

Implementing them anyway, test-first

Spec-legal or not, ControllerPublishVolume and ControllerUnpublishVolume are still two ordinary RPCs with defined request and response shapes, and building them the same test-first way as every other RPC in this book is worth doing — both for the practice, and because a later chapter (csi-sanity, Chapter 9) will call every method a ControllerServer exposes, implemented or not.

ControllerPublishVolumeRequest carries volume_id, node_id, and volume_capability, all three marked REQUIRED by the spec. readonly is also REQUIRED as a field. But it's a plain bool with a usable zero value. There's nothing to validate there the way there is for a string that might be empty, or a pointer that might be nil.

The spec text adds one more rule for node_id: "the CO SHALL set this field to match the node ID returned by NodeGetInfo." Read that rule carefully, because it's easy to misread. It's a promise the CO makes. It's not a check the SP runs. ControllerPublishVolume below does not, and will not, verify that promise — it only checks that node_id isn't empty, the same shallow check every other required string here gets.

That's worth saying plainly, not glossing over: this driver trusts the CO's node_id, it doesn't verify it. And there's a concrete reason it can't easily do better. d.nodeID — the field NodeGetInfo reads from, back in Chapter 5 — only gets set correctly on the Node DaemonSet, where deploy/node.yaml wires up the NODE_NAME Downward API field. The Controller Deployment doesn't set that same env var. So on the Controller pod, Chapter 5's own fallback path kicks in instead — d.nodeID ends up as the pod's own hostname, something like localdir-csi-controller-6f7d8b9c4d-x7k2p, not a real cluster node name. Chapter 5 already explained exactly why that value is wrong for this purpose. Comparing req.GetNodeId() against a value that's already known to be wrong wouldn't add a real check — it would just add a check that's guaranteed to fail. So ControllerPublishVolume doesn't attempt it, and says so honestly instead of pretending to validate something it can't.

It's tempting to add that comparison anyway. Something like this:

if req.GetNodeId() != d.nodeID {
	return nil, status.Errorf(codes.InvalidArgument,
		"driver nodeID %v does not match request nodeID %v", d.nodeID, req.GetNodeId())
}

Don't. Two separate problems, not one.

The first is the concrete one, from just above: on the real Controller pod, d.nodeID isn't a node name. It's a pod hostname. This check would compare a real, correct node_id against a value already known to be wrong. Every publish call would fail — including the exact grpcurl call this chapter runs a few pages from now.

The second is deeper, and worth understanding even outside this one bug. One Controller pod serves the whole cluster, not one node. It has to accept node_id values naming any node out there — not just whichever node it happens to be scheduled on itself. d.nodeID answers a different question: "which node is this process running on." That question only makes sense for the Node service, where one pod really does correspond to one node. Reusing the same field on the Controller side quietly asks it something it was never built to answer. Fixing the pod-hostname issue wouldn't fix this — even a perfectly correct d.nodeID on the Controller side would still only ever equal one node, never many.

A real version of this check would need a real list of known nodes to check against — something like reading Kubernetes's own Node or CSINode objects through the API, which this driver's ServiceAccount already has read access to (Chapter 7's RBAC includes nodes, for external-provisioner's own use) but which ControllerPublishVolume itself never calls. Worth knowing, too: even a driver that does run this check should return NOT_FOUND (per the spec's own error table), not InvalidArgument — a well-formed node_id naming a node that doesn't exist isn't a malformed request, it's a request about something missing.

func TestControllerPublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.ControllerPublishVolumeRequest]{
		{
			name: "missing volume id",
			req: &csi.ControllerPublishVolumeRequest{
				NodeId:           "test-node-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing node id",
			req: &csi.ControllerPublishVolumeRequest{
				VolumeId:         "vol-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing volume capability",
			req: &csi.ControllerPublishVolumeRequest{
				VolumeId: "vol-1",
				NodeId:   "test-node-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.ControllerPublishVolumeRequest) error {
			_, err := d.ControllerPublishVolume(t.Context(), req)
			return err
		},
	)
}

Driver already embeds UnimplementedControllerServer, same as it has since Chapter 7, and that embedded type already has stub ControllerPublishVolume/ControllerUnpublishVolume methods returning Unimplemented — so, unlike Chapter 5 or 7, there's no interface to add and no NewDriver signature to touch this time. No new imports either: validationCase[T] and runValidation are the same generic helpers Chapters 6 and 7 already reached for, and this three-row table is exactly the shape they exist to collapse. The method just needs a real body:

--- FAIL: TestControllerPublishVolume_Validation (0.00s)
    --- FAIL: TestControllerPublishVolume_Validation/missing_volume_id (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: NodeId:test-node-1 VolumeCapability:0xc00000e300 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestControllerPublishVolume_Validation/missing_node_id (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 NodeId: VolumeCapability:0xc00000e318 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestControllerPublishVolume_Validation/missing_volume_capability (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 NodeId:test-node-1 VolumeCapability:<nil> Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

code = Unimplemented, exactly as the paragraph above predicted — that's the embedded stub talking, not this test's own validation. Still red, and still for the right reason: nothing here validates its arguments yet, it just happens to fail with a more specific code than "missing" would. And once again every failure lands on testing_test.go:85, not on this test's own call to runValidation — the same t.Helper() mechanic Chapters 6 and 7 already worked through, holding here without any new explanation needed.

func (d *Driver) ControllerPublishVolume(
	ctx context.Context,
	req *csi.ControllerPublishVolumeRequest,
) (*csi.ControllerPublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetNodeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "node_id is required")
	}
	if req.GetVolumeCapability() == nil {
		return nil, status.Error(codes.InvalidArgument, "volume_capability is required")
	}

	return &csi.ControllerPublishVolumeResponse{}, nil
}

Green — and now the interesting question: what should ControllerPublishVolume actually do, once its arguments pass validation? The spec's error-code table gives a direct answer: NOT_FOUND (5) if "the volume does not exist." A volume genuinely can be published against a volume ID nothing created — a stale claim, a typo, a race — so that's worth its own test, ahead of writing the happy path:

func TestControllerPublishVolume_VolumeNotFound(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "does-not-exist",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	_, err := d.ControllerPublishVolume(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

requireStatusCode takes whatever codes.Code a test expects, not just InvalidArgument — the same helper Chapter 7 wrote fits a NotFound assertion exactly as well as a validation one.

--- FAIL: TestControllerPublishVolume_VolumeNotFound (0.00s)
    controller_test.go:181: code = OK, want NotFound (req=&{VolumeId:does-not-exist NodeId:test-node-1 VolumeCapability:0xc00000e438 Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

This one lands on controller_test.go directly, not testing_test.go — a single direct call to requireStatusCode, the same shape as Chapter 7's TestDeleteVolume_Validation, not one routed through runValidation's closure.

Red, because right now ControllerPublishVolume doesn't check anything beyond its own arguments — it happily reports success for a volume that was never created. The fix reuses a pattern this book has leaned on since NodePublishVolume in Chapter 6: a volume's existence, as far as this driver is concerned, is whether filepath.Join(d.dataDir, volumeID) exists on disk.

	path := filepath.Join(d.dataDir, req.GetVolumeId())
	if _, err := os.Stat(path); os.IsNotExist(err) {
		return nil, status.Errorf(codes.NotFound, "volume %q not found", req.GetVolumeId())
	} else if err != nil {
		return nil, status.Errorf(codes.Internal, "checking volume %q: %v", req.GetVolumeId(), err)
	}

=== RUN   TestControllerPublishVolume_VolumeNotFound
--- PASS: TestControllerPublishVolume_VolumeNotFound (0.00s)

That os.Stat call runs inside the Controller pod's own container. Worth asking directly: whose disk is it actually checking?

In this book's cluster, there's only one possible answer. It's the one kind node. Controller and every Node pod are containers on that same one machine. They share the exact same hostPath, /var/lib/localdir-csi/data — Chapter 7 already pointed this out, when deploy/controller.yaml first mounted it. So this os.Stat check, and NodePublishVolume's own existence check later, in a completely different pod, end up looking at the exact same file on the exact same disk. It only looks like two separate checks.

A real multi-node cluster breaks that. Say the Controller pod lands on node A. CreateVolume, back in Chapter 7, creates the volume's directory on node A's own local disk — wherever Controller happened to be scheduled, not wherever the volume is actually needed. Now say a pod that wants this volume gets scheduled onto node B instead. NodePublishVolume runs inside a Node pod on node B. It checks node B's own local disk. Node A's directory was never there to begin with, and nothing in this driver ever copies data between nodes.

So this os.Stat check depends on the same fact Chapter 7 already named as this driver's "central, load-bearing simplification... why the driver's name starts with 'local.'" That fact doesn't just affect CreateVolume. It quietly reaches into every method that computes filepath.Join(dataDir, volumeID) and treats the result as meaningful — ControllerPublishVolume included. This is the same gap, seen again from a different angle.

Two more cases round this method out, and both go straight to green without any further production code — which is itself worth pausing on, rather than treating as nothing happening. A happy-path test, publishing a volume that genuinely exists:

func TestControllerPublishVolume_Succeeds(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "vol-1",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerPublishVolume() returned an error: %v", err)
	}
}

makeVolumeDir again, same as Chapter 7's TestDeleteVolume_RemovesTheVolume — a fake volume on disk is a fake volume on disk, regardless of which Controller method is about to look for it.

And an idempotency test, calling it twice for the same volume and node:

func TestControllerPublishVolume_Idempotent(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	req := &csi.ControllerPublishVolumeRequest{
		VolumeId:         "vol-1",
		NodeId:           "test-node-1",
		VolumeCapability: mountCapability(),
	}

	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("first ControllerPublishVolume() returned an error: %v", err)
	}
	if _, err := d.ControllerPublishVolume(t.Context(), req); err != nil {
		t.Fatalf("second ControllerPublishVolume() for the same volume/node returned an error: %v", err)
	}
}

Both pass immediately. That's not a wasted test — it's the test proving a specific claim: that a directory-existence check has no state of its own to become inconsistent on a second call. CreateVolume's idempotency back in Chapter 7 needed os.MkdirAll's own already-exists-is-fine behavior; this one gets the same property for free from os.Stat never changing anything. Writing the test first either catches a case where that assumption is wrong, or — as here — turns "I'm pretty sure this is already idempotent" into "a test says so."

ControllerUnpublishVolume is smaller. Its request has only volume_id REQUIRED — node_id is explicitly OPTIONAL here, unlike its counterpart on the publish side, and the spec text explains why, using its own shorthand for "the driver itself," the SP (Storage Plugin) — the other half of the CO/SP pair Chapter 6 already introduced CO for: "If the value is set, the SP MUST unpublish the volume from the specified node. If the value is unset, the SP MUST unpublish the volume from all nodes it is published to." A single-node driver like this one never has to act on that distinction, but the validation has to reflect it correctly regardless — requiring node_id here would reject a spec-legal request.

func TestControllerUnpublishVolume_Validation(t *testing.T) {
	d := newTestDriverInTempDir(t)

	_, err := d.ControllerUnpublishVolume(t.Context(), &csi.ControllerUnpublishVolumeRequest{})
	requireStatusCode(t, err, codes.InvalidArgument, &csi.ControllerUnpublishVolumeRequest{})
}

--- FAIL: TestControllerUnpublishVolume_Validation (0.00s)
    controller_test.go:223: code = Unimplemented, want InvalidArgument (req=&{VolumeId: NodeId: Secrets:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.005s
FAIL

func (d *Driver) ControllerUnpublishVolume(
	ctx context.Context,
	req *csi.ControllerUnpublishVolumeRequest,
) (*csi.ControllerUnpublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}

	return &csi.ControllerUnpublishVolumeResponse{}, nil
}

The spec's idempotency text for unpublish is more forgiving than publish's, on purpose — it says a driver "SHOULD" return 0 OK even when the volume or node "can not be found and can be safely regarded as ControllerUnpublished," rather than the MUST NOT succeed silently tone ControllerPublishVolume takes toward a genuinely missing volume. That asymmetry exists for a real reason: a CO retrying an unpublish call after a partial failure needs "already gone" to look identical to "successfully removed," or cleanup logic ends up stuck re-requesting an unpublish that can never appear to succeed. This driver's implementation already matches that — it doesn't check the volume exists at all, so "volume already gone" and "volume never existed" both fall through to the same 0 OK — which the tests confirm directly, no new production code required for any of them:

func TestControllerUnpublishVolume_Succeeds(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "vol-1",
		NodeId:   "test-node-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() returned an error: %v", err)
	}
}

func TestControllerUnpublishVolume_NodeIDIsOptional(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "vol-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() with no node_id returned an error: %v", err)
	}
}

func TestControllerUnpublishVolume_Idempotent(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.ControllerUnpublishVolumeRequest{
		VolumeId: "already-unpublished",
		NodeId:   "test-node-1",
	}

	if _, err := d.ControllerUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("ControllerUnpublishVolume() for an already-unpublished volume returned an error: %v", err)
	}
}

=== RUN   TestControllerUnpublishVolume_Succeeds
--- PASS: TestControllerUnpublishVolume_Succeeds (0.00s)
=== RUN   TestControllerUnpublishVolume_NodeIDIsOptional
--- PASS: TestControllerUnpublishVolume_NodeIDIsOptional (0.00s)
=== RUN   TestControllerUnpublishVolume_Idempotent
--- PASS: TestControllerUnpublishVolume_Idempotent (0.00s)
PASS

The choice not to advertise any of this

Both methods work, are tested, and would behave correctly if a CO called them. None of that means Kubernetes should ever be told to call them — and here the earlier "USB drive that was never unplugged" framing turns into a concrete set of decisions, each one worth making on purpose rather than by accident.

ControllerGetCapabilities, from Chapter 7, still reports only CREATE_DELETE_VOLUME. It is not getting PUBLISH_UNPUBLISH_VOLUME added now that the RPC exists — the capability list describes what a CO should rely on this driver for, not what methods happen to have a non-stub body. Advertising a capability a driver has no real use for would be actively misleading, not just unnecessary.

That decision connects to a piece of Kubernetes machinery this book hasn't described yet: VolumeAttachment objects. When a CSI driver does need attach/detach, Kubernetes's own in-tree attach/detach controller — not any sidecar container — is what creates a VolumeAttachment object for a volume about to be mounted, and it's external-attacher's job to notice that object and turn it into the actual ControllerPublishVolume gRPC call. Critically, that controller only creates VolumeAttachment objects at all when the driver's own CSIDriver object has spec.attachRequired: true — and Chapter 4 already set attachRequired: false in deploy/csidriver.yaml, for exactly this driver. With that field false, no VolumeAttachment object is ever created, for any volume, ever — which means even if external-attacher were deployed as a third sidecar in deploy/controller.yaml, there would be nothing for it to watch. It would sit there, connected to the same socket as localdir-csi and external-provisioner, doing precisely nothing, forever. Deploying it would cost a container's worth of memory and a line in controller-rbac.yaml in exchange for a permanently idle process — worse than doing nothing, because it would look like this driver needed something it doesn't.

This isn't a guess about how a real driver would handle a "nothing to attach" case — it's how one actually does. csi-driver-nfs, a real, maintained CSI driver for a backend that (like this book's) never needs attaching, sets attachRequired: false, does not report PUBLISH_UNPUBLISH_VOLUME, implements ControllerPublishVolume and ControllerUnpublishVolume as bare Unimplemented stubs, and ships no csi-attacher sidecar in its own manifests at all. localdir-csi's choice here is the same shape, just slightly more thorough — this chapter gave both RPCs real logic instead of leaving them as stubs, purely to show the pattern the way Chapter 4 promised. Contrast that with csi-driver-host-path, a reference driver for storage that does model a real attach step: it leaves attachRequired unset (which defaults to true), does report PUBLISH_UNPUBLISH_VOLUME, and does deploy csi-attacher. Same spec, same RPC signatures, genuinely different deployment — because the backends are genuinely different, and each driver's manifests say so honestly.

Chapter 2's original sketch of deploy/controller.yaml as "driver + provisioner + attacher sidecars" turns out to have been a reasonable early guess that this chapter overturns, and it's worth being direct about that rather than quietly ignoring it: controller.yaml still has exactly the two containers Chapter 7 gave it. Nothing here adds a third.

Proving it, with grpcurl

Same shape as Chapter 7's proof for CreateVolume/DeleteVolume — kubectl cp the same grpcurl-linux binary into the Controller pod, kubectl exec into the localdir-csi container specifically. First, create a real volume to publish, the same way Chapter 7 did:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "attach-demo",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

{
  "volume": {
    "capacityBytes": "1048576",
    "volumeId": "attach-demo"
  }
}

Now publish it — csi-dev-control-plane is this single-node kind cluster's actual node name, the same value NodeGetInfo returns for real, via the NODE_NAME Downward API field Chapter 5 wired up:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo",
  "node_id": "csi-dev-control-plane",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:///csi/csi.sock csi.v1.Controller/ControllerPublishVolume

{}

0 OK, empty response — TestControllerPublishVolume_Succeeds, proved against a real socket. Call it again, identical request:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo",
  "node_id": "csi-dev-control-plane",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:///csi/csi.sock csi.v1.Controller/ControllerPublishVolume

{}

Still 0 OK — TestControllerPublishVolume_Idempotent. Now unpublish, deliberately omitting node_id to exercise the optional case directly:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/ControllerUnpublishVolume

{}

And once more, against a volume ID that's already unpublished:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/ControllerUnpublishVolume

{}

Both 0 OK — the same forgiving idempotency the spec text asked for, now demonstrated against a real driver process instead of just a table of test assertions. Clean up before moving on:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "volume_id": "attach-demo"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteVolume

Part two: closing the loop, for real

Every RPC call in this book so far has been typed by a person into grpcurl. That was never the end goal — it was a way to prove each piece works in isolation before wiring it into the machinery that's supposed to call it automatically. That machinery has been present in the cluster since Chapter 7 (external-provisioner) and Chapter 4/6 (node-driver-registrar, and kubelet itself) — it just hasn't had anything to react to yet, because nothing has created a PersistentVolumeClaim. This section creates one, and nothing about the next thousand words involves grpcurl at all.

StorageClass: telling Kubernetes which driver to ask

A PersistentVolumeClaim doesn't name a driver directly — it names a StorageClass, and the StorageClass names the driver. That indirection is what lets a cluster operator swap which actual storage backend a claim resolves to without every application's YAML needing to know or care.

Create deploy/storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: localdir-csi
provisioner: localdir.csi.example.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

provisioner is the one required field, and it has to match this driver's name exactly — the same string GetPluginInfo has returned since Chapter 3, and the same string external-provisioner already checked for on every PersistentVolumeClaim it's been watching since Chapter 7. reclaimPolicy: Delete (the default, but worth writing explicitly) means the underlying volume — and, per Chapter 7's DeleteVolume, the actual directory on disk — gets deleted automatically once its PersistentVolumeClaim is deleted, rather than left around for an operator to clean up by hand.

volumeBindingMode: Immediate (also the default) deserves an honest caveat rather than a silent pass. Immediate tells external-provisioner to call CreateVolume as soon as a claim appears, before Kubernetes has picked which node any pod using it will run on. That's fine on this book's single-node kind cluster, where every possible node is the same node — but it's a real limitation for a driver that ever runs on more than one machine, since a volume created on Node A does this driver no good for a pod the scheduler puts on Node B. The correct fix for a genuinely multi-node local-storage driver is volumeBindingMode: WaitForFirstConsumer, which delays CreateVolume until a pod actually needs the claim and a node is already chosen, combined with the driver reporting a VOLUME_ACCESSIBILITY_CONSTRAINTS plugin capability and reading CreateVolumeRequest.accessibility_requirements to create the volume in the right place. localdir-csi does neither — it's a real gap, not a hidden one, and exactly the kind of thing "retargeting CreateVolume at a real storage API," the way Chapter 2 originally framed this whole project, would need to revisit.

PersistentVolumeClaim and Pod: the demo

These two objects aren't part of the driver's own deployment the way deploy/controller.yaml or deploy/node.yaml are — they're a workload using the driver, with a lifecycle that has nothing to do with the driver's own. Keep them out of deploy/, which make deploy applies in full every time; a stray demo Pod has no business being recreated on every make deploy run. Create demo-pvc.yaml and demo-pod.yaml in the project root instead:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-claim
spec:
  storageClassName: localdir-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: demo-claim

storageClassName: localdir-csi is the only line in the claim that ties it to this driver at all — everything downstream of it happens because of that one reference. claimName: demo-claim on the Pod side is what turns "a claim exists" into "a pod actually wants to mount it" — Kubernetes won't do anything with an unclaimed PersistentVolumeClaim beyond creating the underlying volume once volumeBindingMode allows it, and with Immediate binding, that happens right away regardless of whether any pod ever references the claim at all.

Apply the claim first, on its own, to watch provisioning happen with nothing else going on:

kubectl apply -f demo-pvc.yaml
kubectl get pvc demo-claim

NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
demo-claim    Pending                                       localdir-csi   2s

Pending — external-provisioner has seen the claim (its watch loop has been running since Chapter 7) but hasn't finished acting on it yet. Check its logs:

kubectl logs -l app=localdir-csi-controller -c external-provisioner --tail=20

I0821 09:12:04.100221       1 controller.go:1449] provision "default/demo-claim" class "localdir-csi": started
I0821 09:12:04.118903       1 controller.go:1546] provision "default/demo-claim" class "localdir-csi": volume "pvc-3f9a2b7e-..." provisioned
I0821 09:12:04.118940       1 controller.go:1563] provision "default/demo-claim" class "localdir-csi": succeeded

That's CreateVolume — the exact same method this book has called by hand with grpcurl since Chapter 7 — being called automatically for the first time in this book, triggered entirely by a kubectl apply. A moment later:

kubectl get pvc demo-claim
kubectl get pv

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
demo-claim    Bound    pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310    1Gi        RWO            localdir-csi   6s

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   AGE
pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310   1Gi        RWO            Delete           Bound    default/demo-claim     localdir-csi   6s

Bound on both sides — external-provisioner's receipt, the PersistentVolume, now formally tied to the claim that asked for it. Now bring in the pod:

kubectl apply -f demo-pod.yaml
kubectl get pod demo-pod

NAME       READY   STATUS    RESTARTS   AGE
demo-pod   1/1     Running   0          9s

kubelet — the same process Chapter 4 first introduced this driver to — called NodePublishVolume on its own here, no grpcurl involved, because a running pod actually referenced the claim. Prove the mount is real, not just reported as ready:

kubectl exec demo-pod -- sh -c 'echo hello from kubernetes > /data/hello.txt'
kubectl exec demo-pod -- cat /data/hello.txt

hello from kubernetes

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/pvc-3f9a2b7e-1c44-4e91-9c02-8f21a7b6d310/hello.txt

hello from kubernetes

Same file, visible from both inside the container and directly on the node's own disk — the same proof-from-both-sides technique Chapter 6 used for NodePublishVolume's very first bind mount, now happening because a pod spec asked for it, not because anyone typed a mount command.

Tearing it down, and watching cleanup happen too

kubectl delete pod demo-pod
kubectl delete pvc demo-claim

Deleting the pod triggers NodeUnpublishVolume (Chapter 6) the same way creating it triggered NodePublishVolume. Deleting the claim, with reclaimPolicy: Delete in effect, triggers something new:

kubectl logs -l app=localdir-csi-controller -c external-provisioner --tail=10

I0821 09:14:51.220118       1 controller.go:1704] delete "pvc-3f9a2b7e-...": started
I0821 09:14:51.231455       1 controller.go:1712] delete "pvc-3f9a2b7e-...": volume deleted

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/

Empty. DeleteVolume, called automatically, removed the same directory grpcurl removed by hand back in Chapter 7 — and the PersistentVolume object itself is gone along with it, not left behind in a Released state, because reclaimPolicy: Delete means "delete," not "release for someone to clean up later."

What you should have now

• internal/driver/controller.go: ControllerPublishVolume and ControllerUnpublishVolume, both real, both tested, neither advertised in ControllerGetCapabilities
• A clear, spec-grounded, real-world-validated reason for that last point: attachRequired: false (set back in Chapter 4) means Kubernetes never creates a VolumeAttachment object for this driver, so external-attacher — undeployed, and correctly so — would have nothing to watch even if it were running
• deploy/storageclass.yaml: a StorageClass naming this driver, with an honestly-flagged Immediate-binding limitation for any future multi-node use
• demo-pvc.yaml and demo-pod.yaml, outside deploy/ on purpose, proving — with zero grpcurl — that a PersistentVolumeClaim alone triggers CreateVolume, that a pod referencing it triggers NodePublishVolume, and that deleting the claim triggers DeleteVolume automatically
• The full loop this book has been building toward since Chapter 1, now running end to end without a single RPC typed by hand

Every RPC this driver implements now has two forms of proof behind it: a unit test with a fake dataDir, and a real kubectl apply exercising the exact same code path through the real Kubernetes control plane. Chapter 9 turns to csi-sanity, the CSI project's own conformance test suite — a way to check this driver against the spec's own expectations directly, instead of only against the specific cases this book happened to think to write.