Chapter 6: The Node Service, Part 2

NodeGetInfo and NodeGetCapabilities answer questions. Neither one touches a filesystem, mounts anything, or changes what a pod can actually see. This chapter writes the two methods that do: NodePublishVolume and NodeUnpublishVolume — the calls that turn a directory of data sitting on a node's disk into something a container can read and write through, and then, later, take that access away again cleanly. This is the payoff Chapter 5 pointed at, and it's also where mount --bind, first mentioned back in Chapter 2, stops being a phrase in a sidebar and becomes something your own code calls.

What "publish" actually means here

A bind mount is a Linux feature that makes one directory also appear at a second path — not a copy, the same files, reachable through two different doors. If /var/lib/localdir-csi/data/vol-1 holds a volume's real data, bind-mounting it onto some other path makes that second path behave exactly like the first: read a file through either one, and you're reading the same bytes.

That's exactly the tool NodePublishVolume needs. Kubernetes decides where a container should see its volume — a path deep inside kubelet's own directory structure, specific to one pod, that your driver doesn't get to choose — and hands you that path as target_path. Your volume's actual data already lives somewhere else, at a location only your driver knows about. A bind mount is how you make the data at your location appear, unmodified, at Kubernetes's location. "Publish" means "make this volume visible at the path you were told to use." "Unpublish" means undo exactly that, leaving the underlying data untouched.

One naming note before the code: the CSI spec itself, and its own generated Go doc comments, refer to whatever's driving these calls as the CO — short for Container Orchestrator. Kubernetes is the only CO this book ever talks to, but the spec is deliberately written so a driver like this one could just as easily plug into a different orchestrator without changing a line of RPC code, which is why "CO" shows up instead of "Kubernetes" in the spec's own text, and in this chapter's code comments.

Chapter 2 already flagged why this can't be tested by just running things on your own laptop: kind nodes are real Linux containers, so when localdir-csi calls mount --bind inside one, it's genuinely mounting something, inside that container's own filesystem — the same kind of operation, on the same kind of system, that would happen on a real cloud VM. That's also why this chapter's hands-on testing happens by reaching into a kind node directly, the way Chapter 2 said it eventually would.

A narrow interface for the one thing we need: Mounter

Performing a real bind mount from Go means calling into k8s.io/mount-utils, the same library Kubernetes's own CSI drivers use — it wraps the underlying mount/umount system calls and the bookkeeping needed to ask "is this path already a mount point?" correctly on Linux. Its Interface type exposes close to a dozen methods (variations on Mount for different flag combinations, List, GetMountRefs, and more). Driver needs exactly three of them: mount something, unmount something, and ask whether a path is currently a mount point.

Rather than depending on the whole real interface, declare a smaller one that says exactly that:

type Mounter interface {
	Mount(source, target, fsType string, options []string) error
	Unmount(target string) error
	IsMountPoint(target string) (bool, error)
}

This is the Interface Segregation Principle, the same family of idea as HealthChecker from Chapter 3: depend on the smallest interface that describes what you actually need, not the largest one a library happens to offer. The payoff here is concrete, not just tidy — because Go decides whether a type satisfies an interface by comparing method signatures, not by any explicit "implements" declaration, the real value k8s.io/mount-utils hands back already has a Mount, an Unmount, and an IsMountPoint with these exact signatures. It satisfies Mounter automatically. No adapter, no wrapper type, no glue code — declaring the interface Driver actually needs is the entire integration.

Driver gains a data directory and a mounter

NodePublishVolume needs to know two new things: where on this node volumes actually live, and something that can perform a mount. Update internal/driver/driver.go:

package driver

import (
	"github.com/container-storage-interface/spec/lib/go/csi"
)

// HealthChecker describes anything that can report whether the driver's
// backend is usable right now. Identity's Probe method depends on this
// interface — not on any specific way of checking health — so that a
// real filesystem check and a test fake are equally valid things to hand
// it. This is the whole idea behind Dependency Inversion: Driver names
// what it needs, not how that need gets satisfied.
type HealthChecker interface {
	Healthy() bool
}

// Mounter describes the three mount operations Driver actually performs.
// A real *mount-utils mounter satisfies this automatically; tests hand it
// a fake that never touches a real filesystem.
type Mounter interface {
	Mount(source, target, fsType string, options []string) error
	Unmount(target string) error
	IsMountPoint(target string) (bool, error)
}

// Driver holds everything our CSI driver needs to answer gRPC calls.
type Driver struct {
	csi.UnimplementedIdentityServer
	csi.UnimplementedNodeServer

	name    string
	version string
	health  HealthChecker
	nodeID  string
	dataDir string
	mount   Mounter
}

// NewDriver builds a Driver. health and mount can be nil — health is
// guarded in Probe, and mount is only touched by RPCs that need it —
// which is useful for tests that don't care about either, but main.go
// always passes real implementations of both.
func NewDriver(name, version string, health HealthChecker, nodeID, dataDir string, mount Mounter) *Driver {
	return &Driver{
		name:    name,
		version: version,
		health:  health,
		nodeID:  nodeID,
		dataDir: dataDir,
		mount:   mount,
	}
}

// Compile-time checks: these lines do nothing at runtime. They exist
// purely so that if you ever change Driver in a way that breaks either
// interface, the build fails immediately with a clear error — instead of
// failing later, mysteriously, when gRPC tries to register your service.
var _ csi.IdentityServer = (*Driver)(nil)
var _ csi.NodeServer = (*Driver)(nil)

NewDriver grew two more parameters, and — the same lesson every constructor change in this book has taught — that breaks every existing call site at once. Run the tests:

go test ./internal/driver/...

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
internal/driver/identity_test.go:10:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/identity_test.go:26:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/identity_test.go:66:97: not enough arguments in call to NewDriver
	have (string, string, *fakeHealthChecker, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/node_test.go:10:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
internal/driver/node_test.go:23:59: not enough arguments in call to NewDriver
	have (string, string, nil, string)
	want (string, string, HealthChecker, string, string, Mounter)
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]
FAIL

None of these five existing tests care about a data directory or a mounter — Identity's tests never touch either, and NodeGetInfo/ NodeGetCapabilities only ever read nodeID. The smallest honest fix would be a placeholder for each — an empty string for dataDir, nil for mount — the same way nil already stands in for a health checker these particular tests don't need. But this is the third time NewDriver has grown a parameter and broken every call site at once, and this chapter is about to add a dozen more tests that all need to build a Driver the same way. Patching five call sites by hand, again, is the smallest fix for today; it isn't the smallest fix for the pattern.

Create internal/driver/testing_test.go — a file with no tests of its own, only shared scaffolding every other test file in this package can call:

package driver

import (
	"testing"
)

// newTestDriver builds a Driver wired to the package's shared test
// constants, with the given dataDir and mount. Pass "" for dataDir when
// a test doesn't need one, and nil for mount when the call path never
// reaches the mounter.
func newTestDriver(t *testing.T, dataDir string, mount Mounter) *Driver {
	t.Helper()
	return NewDriver(name, version, nil, nodeID, dataDir, mount)
}

// newTestDriverInTempDir is the common case: a fresh temp dataDir and no
// mounter. Each call gets its own t.TempDir(), so tests stay isolated
// from each other.
func newTestDriverInTempDir(t *testing.T) *Driver {
	t.Helper()
	return newTestDriver(t, t.TempDir(), nil)
}

newTestDriver hardcodes health to nil — every test that actually needs to control health, like TestProbe, still calls NewDriver directly, exactly as before. What it does fix is the other five parameters: name, version, and nodeID are always the same three values across this entire package, and dataDir/mount are the two that actually vary test to test. t.Helper() matters here for the same reason it always does — a future failure inside a test built this way should point at that test's own line, not at a line inside this helper.

name, version, and nodeID need to actually exist somewhere now. Add this to internal/driver/identity_test.go, alongside its existing imports:

const (
	name    = "localdir.csi.example.com"
	version = "0.1.0"
	nodeID  = "test-node-1"
	dataDir = ""
)

These replace the literal strings scattered across every test file in this package — the same "localdir.csi.example.com", "0.1.0", and node ID that TestGetPluginInfo, TestGetPluginCapabilities, TestProbe, TestNodeGetInfo, and TestNodeGetCapabilities were each already retyping by hand. Update the five call sites to match: the four that don't need a real health checker become

d := newTestDriver(t, dataDir, nil)

and TestProbe's, which still needs to hand Driver a real *fakeHealthChecker, becomes

d := NewDriver(name, version, &fakeHealthChecker{healthy: tt.healthy}, nodeID, dataDir, nil)

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green again — and if NewDriver ever grows another parameter, only newTestDriver itself needs to change. Every test that calls it stays exactly as it is; this is newTestDriver's whole reason for existing.

Writing NodePublishVolume, test-first

Testing this honestly means never touching a real filesystem mount — that needs root, and a test suite that only passes as root isn't one you can trust in CI. Mounter exists precisely so a test can hand Driver a fake that records what it was asked to do instead of doing it. One test at a time, same rhythm as every RPC so far: write a test, watch it fail, write only enough code to pass it, then move to the next case.

Add this to internal/driver/node_test.go, alongside the existing NodeGetInfo/NodeGetCapabilities tests:

type mountCall struct {
	source, target, fsType string
	options                []string
}

type fakeMounter struct {
	mounted         bool
	isMountPointErr error
	mountErr        error
	unmountErr      error

	mountCalls   []mountCall
	unmountCalls []string
}

func (f *fakeMounter) Mount(source, target, fsType string, options []string) error {
	f.mountCalls = append(f.mountCalls, mountCall{source, target, fsType, options})
	return f.mountErr
}

func (f *fakeMounter) Unmount(target string) error {
	f.unmountCalls = append(f.unmountCalls, target)
	return f.unmountErr
}

func (f *fakeMounter) IsMountPoint(target string) (bool, error) {
	if f.isMountPointErr != nil {
		return false, f.isMountPointErr
	}
	return f.mounted, nil
}

fakeMounter belongs in node_test.go — nothing outside Node's own tests needs it. The next piece doesn't: every test below needs one particular *csi.VolumeCapability, and later chapters' Controller tests will need the exact same one. That makes it shared scaffolding, not node_test.go's private business — add it to internal/driver/testing_test.go instead:

func mountCapability() *csi.VolumeCapability {
	return &csi.VolumeCapability{
		AccessType: &csi.VolumeCapability_Mount{Mount: &csi.VolumeCapability_MountVolume{}},
		AccessMode: &csi.VolumeCapability_AccessMode{
			Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
		},
	}
}

Add "github.com/container-storage-interface/spec/lib/go/csi" to testing_test.go's imports.

mountCapability is a test-only helper — nothing in Driver itself calls it — that builds the one piece of a valid request every test below needs and none of them should have to construct by hand: a *csi.VolumeCapability. The CSI spec uses VolumeCapability to describe how a volume needs to be usable, and it's built from two parts. AccessType is a choice between two shapes: a Block capability (the volume shows up as a raw block device, no filesystem at all) or a Mount capability (the volume shows up as a directory you mount). This driver only ever deals in directories, so the helper always builds the Mount variant, and leaves the nested MountVolume value empty — it has fields for a filesystem type and mount flags, neither of which apply here, since every mount this driver performs is a plain bind mount rather than a real filesystem format.

AccessMode describes how many things are allowed to use the volume at once, and how. SINGLE_NODE_WRITER means one node, read-write — the simplest case, and the only one NodePublishVolume assumes right now. The spec defines several other modes, including read-only and multiple-nodes-at-once variants, but nothing in this chapter's code inspects AccessMode's actual value yet: the validation check below only asks whether the VolumeCapability pointer is nil at all, since volume_capability is a required field of the request regardless of which mode it names. AccessMode starts to matter once a real StorageClass and PersistentVolumeClaim can specify one.

This is also the first RPC in the book that needs more than one invalid-request case checked the same way — and it won't be the last; NodeUnpublishVolume, a few pages from now, needs the same shape of test, and so will several Controller methods in Chapter 7. Rather than writing "build a table, loop over it, assert InvalidArgument" by hand every time, add two small pieces of shared scaffolding to internal/driver/testing_test.go:

// requireStatusCode asserts that err carries the given gRPC code. req is
// included in the failure message for debugging.
func requireStatusCode(t *testing.T, err error, want codes.Code, req any) {
	t.Helper()
	if got := status.Code(err); got != want {
		t.Fatalf("code = %v, want %v (req=%+v)", got, want, req)
	}
}

// validationCase is one row of a table-driven validation test: a request
// that should be rejected with codes.InvalidArgument.
type validationCase[T any] struct {
	name string
	req  T
}

// runValidation runs a table of validation cases against call. For each
// case it builds a fresh driver via newDriver, invokes call, and asserts
// the error is InvalidArgument. newDriver is called inside the subtest,
// so per-test temp dirs and contexts stay scoped to that subtest.
func runValidation[T any](
	t *testing.T,
	cases []validationCase[T],
	newDriver func(t *testing.T) *Driver,
	call func(t *testing.T, d *Driver, req T) error,
) {
	t.Helper()
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			d := newDriver(t)
			err := call(t, d, tc.req)
			requireStatusCode(t, err, codes.InvalidArgument, tc.req)
		})
	}
}

Add "google.golang.org/grpc/codes" and "google.golang.org/grpc/status" to testing_test.go's imports.

requireStatusCode replaces the one-off if status.Code(err) != want { t.Fatalf(...) } block every RPC's tests were about to start repeating. validationCase/runValidation go one step further: [T any] is a Go generic — this pair works for *csi.NodePublishVolumeRequest, *csi.NodeUnpublishVolumeRequest, or any other request type, without writing a new loop for each. That's the same Open/Closed shape Chapter 3 already applied to a slice of capabilities, applied here to a slice of test cases instead: a new RPC's validation test becomes a new table and a two-line call to runValidation, not a new copy of the loop.

With that in place, TestNodePublishVolume_Validation itself gets much smaller. Add this to internal/driver/node_test.go, alongside the existing NodeGetInfo/NodeGetCapabilities tests:

func TestNodePublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.NodePublishVolumeRequest]{
		{
			name: "missing volume ID",
			req: &csi.NodePublishVolumeRequest{
				TargetPath:       "/target",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing target path",
			req: &csi.NodePublishVolumeRequest{
				VolumeId:         "vol-1",
				VolumeCapability: mountCapability(),
			},
		},
		{
			name: "missing volume capability",
			req: &csi.NodePublishVolumeRequest{
				VolumeId:   "vol-1",
				TargetPath: "/target",
			},
		},
	}
	runValidation(t, cases,
		func(t *testing.T) *Driver { return newTestDriver(t, dataDir, &fakeMounter{}) },
		func(t *testing.T, d *Driver, req *csi.NodePublishVolumeRequest) error {
			_, err := d.NodePublishVolume(t.Context(), req)
			return err
		},
	)
}

This needs google.golang.org/grpc/codes in node_test.go's imports too — runValidation takes care of status, but node_test.go still names codes.NotFound and friends directly in tests further down (path/filepath and os are coming, but not yet).

Run it:

go test ./internal/driver/...

Red — and a different shape of red than any RPC so far. NodePublishVolume doesn't exist as a method on *Driver yet, so these calls fall through to the embedded UnimplementedNodeServer's stub, which returns a real gRPC status error carrying codes.Unimplemented — not the code this test wants, InvalidArgument, but a real, specific one all the same, not a generic failure. Notice the failure points at testing_test.go, not node_test.go — requireStatusCode calls t.Helper(), but the subtest closure inside runValidation that calls it doesn't, so Go attributes the failure to the innermost non-helper frame, which lives in testing_test.go:

--- FAIL: TestNodePublishVolume_Validation (0.00s)
    --- FAIL: TestNodePublishVolume_Validation/missing_volume_ID (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: PublishContext:map[] StagingTargetPath: TargetPath:/target VolumeCapability:0xc00010c1e0 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestNodePublishVolume_Validation/missing_target_path (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath: VolumeCapability:0xc00010c1f8 Readonly:false Secrets:map[] VolumeContext:map[]})
    --- FAIL: TestNodePublishVolume_Validation/missing_volume_capability (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath:/target VolumeCapability:<nil> Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

(The exact line number inside testing_test.go, and the exact pointer address printed for VolumeCapability, will differ slightly depending on how you've laid the file out — neither one is the point; the code and the message are.)

Write just enough of NodePublishVolume to fix it. status.Error is new this chapter — every RPC before now either succeeded or fell through to UnimplementedNodeServer's stub, never returning a specific failure code on purpose. The CSI spec ties specific conditions to specific gRPC status codes (InvalidArgument for a malformed request), and status.Error(code, message) is how a handler attaches a real one instead of a generic error. Create internal/driver/node.go's new method:

func (d *Driver) NodePublishVolume(
	ctx context.Context,
	req *csi.NodePublishVolumeRequest,
) (*csi.NodePublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetTargetPath() == "" {
		return nil, status.Error(codes.InvalidArgument, "target_path is required")
	}
	if req.GetVolumeCapability() == nil {
		return nil, status.Error(codes.InvalidArgument, "volume_capability is required")
	}

	return &csi.NodePublishVolumeResponse{}, nil
}

That last line is deliberately a stub — the three checks are all this step needs, and returning success unconditionally afterward is the smallest thing that makes the current test pass. Add "google.golang.org/grpc/codes" and "google.golang.org/grpc/status" to node.go's imports too.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.003s

Green. Next case: a volume that doesn't exist. Add this test below TestNodePublishVolume_Validation:

func TestNodePublishVolume_VolumeNotFound(t *testing.T) {
	d := newTestDriver(t, t.TempDir(), &fakeMounter{})

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "does-not-exist",
		TargetPath:       filepath.Join(t.TempDir(), "target"),
		VolumeCapability: mountCapability(),
	}

	_, err := d.NodePublishVolume(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

Add "path/filepath" to node_test.go's imports.

go test ./internal/driver/...

Red — the stub from the last step returns success unconditionally, so there's no NotFound to find yet. This time requireStatusCode is called directly from TestNodePublishVolume_VolumeNotFound itself, no subtest closure in between, so the failure points right back at node_test.go, at requireStatusCode's own call site:

--- FAIL: TestNodePublishVolume_VolumeNotFound (0.00s)
    node_test.go:113: code = OK, want NotFound (req=&{VolumeId:does-not-exist PublishContext:map[] StagingTargetPath: TargetPath:/tmp/TestNodePublishVolume_VolumeNotFound2786629652/002/target VolumeCapability:0xc00010c2e8 Readonly:false Secrets:map[] VolumeContext:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.005s
FAIL

(As ever, the exact temp path and pointer address in your own output will differ from this one.)

Fix it. This is where the source path convention this chapter already described gets written down in code: filepath.Join(d.dataDir, req.GetVolumeId()) assumes a volume's data lives in a directory named after its own ID, directly under the driver's data root — right now that directory has to exist already, or the request fails with NotFound; Chapter 7's CreateVolume is what will make it exist automatically, using this exact same naming rule.

	source := filepath.Join(d.dataDir, req.GetVolumeId())
	if _, err := os.Stat(source); err != nil {
		if os.IsNotExist(err) {
			return nil, status.Errorf(codes.NotFound, "volume %q not found", req.GetVolumeId())
		}
		return nil, status.Errorf(codes.Internal, "checking volume %q: %v", req.GetVolumeId(), err)
	}

	return &csi.NodePublishVolumeResponse{}, nil

That replaces the old stub return — insert it right after the three validation checks in NodePublishVolume, and add "os" and "path/filepath" to node.go's imports.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green. Next: actually mounting something, for a volume that does exist — TestNodePublishVolume_MountsTheVolume, added below TestNodePublishVolume_VolumeNotFound. Setting up "a fake volume directory" is also something every remaining test in this file, and every Controller test Chapter 7 adds, needs to do at least once. Add one more helper to internal/driver/testing_test.go first:

// makeVolumeDir creates a fake volume directory named volName inside
// dataDir and returns its path. Fails the test immediately if the mkdir
// fails.
func makeVolumeDir(t *testing.T, dataDir, volName string) string {
	t.Helper()
	p := filepath.Join(dataDir, volName)
	if err := os.Mkdir(p, 0750); err != nil {
		t.Fatalf("creating fake volume dir %q: %v", p, err)
	}
	return p
}

Add "os" and "path/filepath" to testing_test.go's imports.

func TestNodePublishVolume_MountsTheVolume(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	mounter := &fakeMounter{}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	if _, err := os.Stat(targetPath); err != nil {
		t.Fatalf("target path was not created: %v", err)
	}

	if len(mounter.mountCalls) != 1 {
		t.Fatalf("got %d Mount calls, want 1", len(mounter.mountCalls))
	}
	got := mounter.mountCalls[0]
	wantSource := filepath.Join(dataDir, "vol-1")
	if got.source != wantSource {
		t.Errorf("source = %q, want %q", got.source, wantSource)
	}
	if got.target != targetPath {
		t.Errorf("target = %q, want %q", got.target, targetPath)
	}
	if len(got.options) != 1 || got.options[0] != "bind" {
		t.Errorf("options = %v, want [bind]", got.options)
	}
}

Add "os" to node_test.go's imports too.

go test ./internal/driver/...

Red — NodePublishVolume finds the volume now, but still just returns success without creating the target path or calling Mount at all (t.TempDir() generates a fresh, uniquely-named directory on every run, so the exact path in your own output will differ from this one):

--- FAIL: TestNodePublishVolume_MountsTheVolume (0.00s)
    node_test.go:135: target path was not created: stat /tmp/TestNodePublishVolume_MountsTheVolume3795384414/002/target: no such file or directory
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it — the spec makes creating target_path the driver's job, not the CO's, so NodePublishVolume does that unconditionally before touching the mounter:

	// The spec makes creating target_path our job, not the CO's.
	if err := os.MkdirAll(req.GetTargetPath(), 0750); err != nil {
		return nil, status.Errorf(codes.Internal, "creating target path: %v", err)
	}

	if err := d.mount.Mount(source, req.GetTargetPath(), "", []string{"bind"}); err != nil {
		return nil, status.Errorf(codes.Internal, "mounting volume %q: %v", req.GetVolumeId(), err)
	}

	return &csi.NodePublishVolumeResponse{}, nil

Replace the old return &csi.NodePublishVolumeResponse{}, nil that followed the NotFound check with this.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.004s

Green. Next: honoring readonly. Add this below TestNodePublishVolume_MountsTheVolume:

func TestNodePublishVolume_Readonly(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	mounter := &fakeMounter{}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         true,
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	got := mounter.mountCalls[0].options
	if len(got) != 2 || got[0] != "bind" || got[1] != "ro" {
		t.Errorf("options = %v, want [bind ro]", got)
	}
}

go test ./internal/driver/...

Red — the mount options are hardcoded to [bind] and never look at req.GetReadonly():

--- FAIL: TestNodePublishVolume_Readonly (0.00s)
    node_test.go:175: options = [bind], want [bind ro]
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it — build the options slice instead of hardcoding it:

	options := []string{"bind"}
	if req.GetReadonly() {
		options = append(options, "ro")
	}
	if err := d.mount.Mount(source, req.GetTargetPath(), "", options); err != nil {
		return nil, status.Errorf(codes.Internal, "mounting volume %q: %v", req.GetVolumeId(), err)
	}

This replaces the previous Mount call, which passed []string{"bind"} directly.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green. One case left for this method: a target that's already published. Add this below TestNodePublishVolume_Readonly:

func TestNodePublishVolume_AlreadyMounted(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := t.TempDir()

	mounter := &fakeMounter{mounted: true}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
	}

	if _, err := d.NodePublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodePublishVolume() returned an error: %v", err)
	}

	if len(mounter.mountCalls) != 0 {
		t.Errorf("got %d Mount calls, want 0 — already-mounted target must not be remounted", len(mounter.mountCalls))
	}
}

go test ./internal/driver/...

Red — nothing so far ever asks the mounter whether the target is already mounted, so Mount gets called again even though fakeMounter.mounted is true:

--- FAIL: TestNodePublishVolume_AlreadyMounted (0.00s)
    node_test.go:198: got 1 Mount calls, want 0 — already-mounted target must not be remounted
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it by checking first, and short-circuiting if the answer is yes:

	mounted, err := d.mount.IsMountPoint(req.GetTargetPath())
	if err != nil {
		return nil, status.Errorf(codes.Internal, "checking target path: %v", err)
	}
	if mounted {
		// NodePublishVolume must be idempotent: a CO that isn't sure a
		// previous call succeeded is expected to retry it, and retrying
		// must not fail or remount.
		return &csi.NodePublishVolumeResponse{}, nil
	}

Insert this right after the os.MkdirAll call and before building the options slice.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green — and the common case is handled: the exact same request, retried. The comment above is worth pausing on: idempotent means calling an operation more than once, with the same arguments, has the same effect as calling it exactly once — nothing extra happens, and nothing errors out, the second time. That's not incidental here; the CSI spec requires it, because a CO that times out waiting for a response has no way to know whether the call actually succeeded, and the only safe thing for it to do is call again.

That phrase "with the same arguments" is doing more work than it looks like it is.

Writing NodePublishVolume, test-first: telling a retry from a conflict

IsMountPoint(target_path) only ever answers true or false. It says nothing about what is mounted there, or how. Call NodePublishVolume once for a volume with readonly: false, then again for the same volume and target path with readonly: true, and the check just written waves the second call through anyway — "already mounted" reads as "safe to report success," full stop, regardless of whether this is genuinely the same request retried or a different request that happens to reuse the same volume and target path. Not a hypothetical, either: two Pods mounting the same PVC with different access modes would trigger exactly this.

Add the test, right after TestNodePublishVolume_AlreadyMounted in internal/driver/node_test.go:

func TestNodePublishVolume_RejectsAccessModeMismatch(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	targetPath := filepath.Join(t.TempDir(), "target")

	first := newTestDriver(t, dataDir, &fakeMounter{})
	firstReq := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         false,
	}
	if _, err := first.NodePublishVolume(t.Context(), firstReq); err != nil {
		t.Fatalf("first NodePublishVolume() returned an error: %v", err)
	}

	// A second driver instance, same dataDir, mounter pre-set to report
	// the target as already mounted — the same signal a CO's retry would
	// see from a real kubelet, this time asking for it read-only.
	second := newTestDriver(t, dataDir, &fakeMounter{mounted: true})
	secondReq := &csi.NodePublishVolumeRequest{
		VolumeId:         "vol-1",
		TargetPath:       targetPath,
		VolumeCapability: mountCapability(),
		Readonly:         true,
	}
	_, err := second.NodePublishVolume(t.Context(), secondReq)
	requireStatusCode(t, err, codes.AlreadyExists, secondReq)
}

go test ./internal/driver/...

Red — nothing anywhere records what a mount was actually made with, so there's nothing to compare the second call against:

=== RUN   TestNodePublishVolume_RejectsAccessModeMismatch
    node_test.go:229: code = OK, want AlreadyExists (req=&{VolumeId:vol-1 PublishContext:map[] StagingTargetPath: TargetPath:/tmp/TestNodePublishVolume_RejectsAccessModeMismatch4183913702/002/target VolumeCapability:0xc0000d0090 Readonly:true Secrets:map[] VolumeContext:map[]})
--- FAIL: TestNodePublishVolume_RejectsAccessModeMismatch (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.002s
FAIL

Fixing it means answering a question the driver currently has no way to ask: what was this volume actually published with, last time? A boolean "is something mounted here" isn't enough — the driver needs to remember the actual parameters. That means writing something down: a small metadata file, sitting next to the volume's data but never inside it, recording what a call actually did so a later call can compare against that record instead of guessing from a side effect. Add this to internal/driver/node.go:

// nodePublishMetaPath is where NodePublishVolume records which target path
// and access mode a volume was last published with on this node, so a
// later NodePublishVolume call for the same volume can tell a genuine
// idempotent retry (same target path, same readonly flag) apart from a
// real conflict (a different one).
func nodePublishMetaPath(dataDir, volumeID string) string {
	return filepath.Join(dataDir, ".csi-meta", "publish", volumeID+".json")
}

type nodePublishMeta struct {
	TargetPath string `json:"target_path"`
	Readonly   bool   `json:"readonly"`
}

func writeNodePublishMeta(dataDir, volumeID string, meta nodePublishMeta) error {
	metaBytes, err := json.Marshal(meta)
	if err != nil {
		return err
	}
	metaPath := nodePublishMetaPath(dataDir, volumeID)
	if err := os.MkdirAll(filepath.Dir(metaPath), 0o750); err != nil {
		return err
	}
	return os.WriteFile(metaPath, metaBytes, 0o640)
}

func readNodePublishMeta(dataDir, volumeID string) (nodePublishMeta, error) {
	var meta nodePublishMeta
	data, err := os.ReadFile(nodePublishMetaPath(dataDir, volumeID))
	if err != nil {
		return meta, err
	}
	err = json.Unmarshal(data, &meta)
	return meta, err
}

Add "encoding/json" to node.go's imports. Then change NodePublishVolume's already-mounted branch, in the same file:

	if mounted {
		// NodePublishVolume must be idempotent: a CO that isn't sure a
		// previous call succeeded is expected to retry it, and retrying
		// must not fail or remount. But "already mounted" only means
		// "safe to report success" if it was mounted with these same
		// parameters — otherwise this is a real conflict, not a retry.
		if existing, err := readNodePublishMeta(d.dataDir, req.GetVolumeId()); err == nil {
			if existing.TargetPath != req.GetTargetPath() || existing.Readonly != req.GetReadonly() {
				return nil, status.Errorf(codes.AlreadyExists, "volume %q is already published with different parameters", req.GetVolumeId())
			}
		} else if !os.IsNotExist(err) {
			return nil, status.Errorf(codes.Internal, "reading volume %q publish metadata: %v", req.GetVolumeId(), err)
		}
		return &csi.NodePublishVolumeResponse{}, nil
	}

And record the metadata right after a real mount succeeds, before NodePublishVolume's final return:

	if err := writeNodePublishMeta(d.dataDir, req.GetVolumeId(), nodePublishMeta{
		TargetPath: req.GetTargetPath(),
		Readonly:   req.GetReadonly(),
	}); err != nil {
		return nil, status.Errorf(codes.Internal, "recording volume %q publish metadata: %v", req.GetVolumeId(), err)
	}

NodeUnpublishVolume, written a little further down this chapter, gets one more line of its own to clean this metadata file back up — covered when that method's fix comes up.

go test ./internal/driver/...

=== RUN   TestNodePublishVolume_RejectsAccessModeMismatch
--- PASS: TestNodePublishVolume_RejectsAccessModeMismatch (0.00s)

Green — and now NodePublishVolume is done, in the stronger sense the spec actually asks for. Worth naming the pattern explicitly, because it's going to come up again: an operation that's supposed to be idempotent is not automatically safe just because the primitive underneath it happens not to fail on a repeat call — IsMountPoint returning true here, but the same shape shows up anywhere a driver checks "does this already exist" instead of "does this already exist as this specific thing." "Looks idempotent" and "is idempotent" are different claims. The first is free. The second costs one metadata file and one comparison, every time: record what you actually did, compare what's being asked against what you recorded, and only then decide whether this is the same request again or a different one wearing the same name. Keep an eye out for this shape — CreateVolume in Chapter 7 is about to skip it, on purpose, and pay for that choice in Chapter 9.

Writing NodeUnpublishVolume, test-first

Same rhythm, new method. Start with validation, the same shape as NodePublishVolume's first test. Add this below TestNodePublishVolume_AlreadyMounted:

func TestNodeUnpublishVolume_Validation(t *testing.T) {
	cases := []validationCase[*csi.NodeUnpublishVolumeRequest]{
		{name: "missing volume ID", req: &csi.NodeUnpublishVolumeRequest{TargetPath: "/target"}},
		{name: "missing target path", req: &csi.NodeUnpublishVolumeRequest{VolumeId: "vol-1"}},
	}
	runValidation(t, cases,
		func(t *testing.T) *Driver { return newTestDriver(t, dataDir, &fakeMounter{}) },
		func(t *testing.T, d *Driver, req *csi.NodeUnpublishVolumeRequest) error {
			_, err := d.NodeUnpublishVolume(t.Context(), req)
			return err
		},
	)
}

go test ./internal/driver/...

Red — NodeUnpublishVolume is still only the promoted UnimplementedNodeServer stub, the same Unimplemented-code shape NodePublishVolume's first test hit. And the same testing_test.go attribution as before, for the same reason:

--- FAIL: TestNodeUnpublishVolume_Validation (0.00s)
    --- FAIL: TestNodeUnpublishVolume_Validation/missing_volume_ID (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId: TargetPath:/target})
    --- FAIL: TestNodeUnpublishVolume_Validation/missing_target_path (0.00s)
        testing_test.go:85: code = Unimplemented, want InvalidArgument (req=&{VolumeId:vol-1 TargetPath:})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Write just enough to fix it — the same two-check, stub-return shape NodePublishVolume started with:

func (d *Driver) NodeUnpublishVolume(
	ctx context.Context,
	req *csi.NodeUnpublishVolumeRequest,
) (*csi.NodeUnpublishVolumeResponse, error) {
	if req.GetVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "volume_id is required")
	}
	if req.GetTargetPath() == "" {
		return nil, status.Error(codes.InvalidArgument, "target_path is required")
	}

	return &csi.NodeUnpublishVolumeResponse{}, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.005s

Green. Next: actually unmounting and cleaning up. Add this below TestNodeUnpublishVolume_Validation:

func TestNodeUnpublishVolume_UnmountsAndRemoves(t *testing.T) {
	targetPath := filepath.Join(t.TempDir(), "target")
	if err := os.Mkdir(targetPath, 0750); err != nil {
		t.Fatalf("setting up fake target path: %v", err)
	}

	mounter := &fakeMounter{mounted: true}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodeUnpublishVolumeRequest{
		VolumeId:   "vol-1",
		TargetPath: targetPath,
	}

	if _, err := d.NodeUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodeUnpublishVolume() returned an error: %v", err)
	}

	if len(mounter.unmountCalls) != 1 || mounter.unmountCalls[0] != targetPath {
		t.Errorf("unmountCalls = %v, want [%q]", mounter.unmountCalls, targetPath)
	}

	if _, err := os.Stat(targetPath); !os.IsNotExist(err) {
		t.Errorf("target path still exists after NodeUnpublishVolume")
	}
}

go test ./internal/driver/...

Red — the stub from the last step returns success without ever calling Unmount or removing anything (again, the exact temp path in your output will be different from this one):

--- FAIL: TestNodeUnpublishVolume_UnmountsAndRemoves (0.00s)
    node_test.go:235: unmountCalls = [], want ["/tmp/TestNodeUnpublishVolume_UnmountsAndRemoves1008210749/001/target"]
    node_test.go:239: target path still exists after NodeUnpublishVolume
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

Fix it. The CSI spec's own text for this method says the driver "MUST delete the file or directory it created" at target_path — the os.RemoveAll below isn't tidiness, it's a spec requirement:

	if err := d.mount.Unmount(req.GetTargetPath()); err != nil {
		return nil, status.Errorf(codes.Internal, "unmounting %q: %v", req.GetTargetPath(), err)
	}

	if err := os.RemoveAll(req.GetTargetPath()); err != nil {
		return nil, status.Errorf(codes.Internal, "removing target path: %v", err)
	}

	// Clean up the publish metadata NodePublishVolume recorded, so a
	// volume republished later starts with no stale record of what an
	// earlier, now-unpublished call did.
	if err := os.Remove(nodePublishMetaPath(d.dataDir, req.GetVolumeId())); err != nil && !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "deleting volume %q publish metadata: %v", req.GetVolumeId(), err)
	}

	return &csi.NodeUnpublishVolumeResponse{}, nil

This replaces the old return &csi.NodeUnpublishVolumeResponse{}, nil stub, right after the two validation checks — and closes the loop on the metadata file NodePublishVolume started writing earlier in this chapter.

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.007s

Green. Last case: idempotency, the same property NodePublishVolume needed. Add this below TestNodeUnpublishVolume_UnmountsAndRemoves:

func TestNodeUnpublishVolume_NotMounted(t *testing.T) {
	mounter := &fakeMounter{isMountPointErr: os.ErrNotExist}
	d := newTestDriver(t, dataDir, mounter)

	req := &csi.NodeUnpublishVolumeRequest{
		VolumeId:   "vol-1",
		TargetPath: filepath.Join(t.TempDir(), "already-gone"),
	}

	if _, err := d.NodeUnpublishVolume(t.Context(), req); err != nil {
		t.Fatalf("NodeUnpublishVolume() returned an error: %v", err)
	}

	if len(mounter.unmountCalls) != 0 {
		t.Errorf("got %d Unmount calls, want 0 — nothing was mounted", len(mounter.unmountCalls))
	}
}

go test ./internal/driver/...

Red — NodeUnpublishVolume still calls Unmount unconditionally, so it does even when the fake reports the target was never there in the first place:

--- FAIL: TestNodeUnpublishVolume_NotMounted (0.00s)
    node_test.go:257: got 1 Unmount calls, want 0 — nothing was mounted
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.003s
FAIL

Fix it the same way NodePublishVolume learned to ask first:

	mounted, err := d.mount.IsMountPoint(req.GetTargetPath())
	if err != nil {
		if os.IsNotExist(err) {
			// Nothing left to clean up — this is what a retried call
			// after a previous, fully-successful one looks like.
			return &csi.NodeUnpublishVolumeResponse{}, nil
		}
		return nil, status.Errorf(codes.Internal, "checking target path: %v", err)
	}

	if mounted {
		if err := d.mount.Unmount(req.GetTargetPath()); err != nil {
			return nil, status.Errorf(codes.Internal, "unmounting %q: %v", req.GetTargetPath(), err)
		}
	}

Replace the unconditional Unmount call with this — the os.RemoveAll call that follows stays exactly as it was.

go test ./internal/driver/... -v

=== RUN   TestGetPluginInfo
--- PASS: TestGetPluginInfo (0.00s)
=== RUN   TestGetPluginCapabilities
--- PASS: TestGetPluginCapabilities (0.00s)
=== RUN   TestProbe
=== RUN   TestProbe/backend_is_healthy
=== RUN   TestProbe/backend_is_unhealthy
--- PASS: TestProbe (0.00s)
    --- PASS: TestProbe/backend_is_healthy (0.00s)
    --- PASS: TestProbe/backend_is_unhealthy (0.00s)
=== RUN   TestNodeGetInfo
--- PASS: TestNodeGetInfo (0.00s)
=== RUN   TestNodeGetCapabilities
--- PASS: TestNodeGetCapabilities (0.00s)
=== RUN   TestNodePublishVolume_Validation
=== RUN   TestNodePublishVolume_Validation/missing_volume_ID
=== RUN   TestNodePublishVolume_Validation/missing_target_path
=== RUN   TestNodePublishVolume_Validation/missing_volume_capability
--- PASS: TestNodePublishVolume_Validation (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_volume_ID (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_target_path (0.00s)
    --- PASS: TestNodePublishVolume_Validation/missing_volume_capability (0.00s)
=== RUN   TestNodePublishVolume_VolumeNotFound
--- PASS: TestNodePublishVolume_VolumeNotFound (0.00s)
=== RUN   TestNodePublishVolume_MountsTheVolume
--- PASS: TestNodePublishVolume_MountsTheVolume (0.00s)
=== RUN   TestNodePublishVolume_Readonly
--- PASS: TestNodePublishVolume_Readonly (0.00s)
=== RUN   TestNodePublishVolume_AlreadyMounted
--- PASS: TestNodePublishVolume_AlreadyMounted (0.00s)
=== RUN   TestNodeUnpublishVolume_Validation
=== RUN   TestNodeUnpublishVolume_Validation/missing_volume_ID
=== RUN   TestNodeUnpublishVolume_Validation/missing_target_path
--- PASS: TestNodeUnpublishVolume_Validation (0.00s)
    --- PASS: TestNodeUnpublishVolume_Validation/missing_volume_ID (0.00s)
    --- PASS: TestNodeUnpublishVolume_Validation/missing_target_path (0.00s)
=== RUN   TestNodeUnpublishVolume_UnmountsAndRemoves
--- PASS: TestNodeUnpublishVolume_UnmountsAndRemoves (0.00s)
=== RUN   TestNodeUnpublishVolume_NotMounted
--- PASS: TestNodeUnpublishVolume_NotMounted (0.00s)
PASS
ok  	github.com/yourname/localdir-csi/internal/driver	0.007s

Green, and every test written across both methods still passes — eight small red/green cycles, none of them touching a real filesystem mount, adding up to the two methods that let a pod actually use a volume.

Letting the container actually mount things

A working Mounter is only half the story — the container it runs in needs the mount command itself to actually exist, needs to be allowed to call it, and needs to be able to reach the exact host path kubelet expects a volume to land at. All three are easy to get wrong silently.

k8s.io/mount-utils's Mount doesn't perform a bind mount through a Linux system call directly — it shells out to a real mount executable, the same one you'd run by typing mount at a terminal, resolved from whatever $PATH the process sees. The distroless image Chapter 4 built the final container stage from was chosen because it has nothing beyond the compiled binary and the C library it links against — no shell, no coreutils, no mount. That never mattered before this chapter, since nothing earlier ever shelled out to anything, but NodePublishVolume fails the moment it's asked to mount something for real:

mounting volume "test-vol-1": mount failed: exec: "mount": executable file not found in $PATH

Fix it by building the final image from something that actually ships a mount binary. Alpine's package manager makes this a small, deliberate addition rather than an all-or-nothing tradeoff — util-linux is the package that provides a real mount:

# Stage 2: the image that actually ships. Alpine is still small — a few
# megabytes, not a full distro — but unlike the fully binary-less
# distroless image, it can host the one external command NodePublishVolume
# genuinely needs: a real mount.
FROM alpine:3.24
RUN apk add --no-cache util-linux
COPY --from=build /out/localdir-csi /localdir-csi
ENTRYPOINT ["/localdir-csi"]

This replaces the Dockerfile's second stage from Chapter 4 — everything from FROM gcr.io/distroless/static-debian12 onward — while the first stage, the one that actually compiles the binary, is unchanged. That first stage's CGO_ENABLED=0 flag is exactly why the swap is safe: it produces a binary with no dynamic C library dependency at all, so it runs identically whichever C library the final image happens to carry — glibc, in distroless, or musl, in Alpine. csi-driver-host-path, the reference CSI implementation this chapter has already leaned on for real-world detail, makes the identical choice for the identical reason: its own node plugin builds on Alpine with util-linux installed, because a driver that calls mount needs mount to exist somewhere inside its own container.

Linux treats mounting — even a bind mount between two directories the calling process already owns — as a privileged operation, gated behind a capability ordinary containers don't have by default. Without it, d.mount.Mount(...) returns a permission error the instant it runs for real, regardless of how correct the surrounding Go code is. Grant it by adding a securityContext to the localdir-csi container:

        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

Separately, target_path lives under /var/lib/kubelet/pods/ on the node — kubelet's own directory, not one your driver controls, and not one it currently has any access to at all. It needs a hostPath mount for that directory too, the same pattern Chapter 4 used for the socket directory. But a plain hostPath mount isn't quite enough here, and it's worth understanding why.

Every container gets its own private view of what's currently mounted where — a mount namespace. By default, a mount --bind your driver performs inside its own container stays inside that container's mount namespace; kubelet, watching the real host filesystem from outside any container, would never see it happen. mountPropagation: Bidirectional on a volume mount widens that in both directions: mounts made inside the container become visible on the host (and to other containers sharing the mount), and mounts made on the host become visible inside the container. Without it, NodePublishVolume would report success while kubelet's own view of the filesystem never changed — a working RPC call producing an invisible result.

Add both the volume mount and the corresponding volume:

          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional

        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory

type: Directory, not DirectoryOrCreate — like registration-dir in Chapter 4, this is a path kubelet itself manages and already creates on every node it runs on; if it's missing, something more fundamental than this manifest is wrong.

The full, current deploy/node.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: localdir-csi-node
  labels:
    app: localdir-csi
spec:
  selector:
    matchLabels:
      app: localdir-csi
  template:
    metadata:
      labels:
        app: localdir-csi
    spec:
      containers:
        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/localdir.csi.example.com/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: data-dir
          hostPath:
            path: /var/lib/localdir-csi/data
            type: DirectoryOrCreate
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory

Wiring up a real Mounter in main.go

internal/driver's tests never need a real mounter, but main.go does. Pull in the library:

go get k8s.io/mount-utils

k8s.io/mount-utils declares in its own go.mod that it needs Go 1.26 or newer, and Go's module system won't let a module depend on something that requires a newer Go version than it declares itself — so this command also raises go.mod's own go line to 1.26.0. On your own machine that change is invisible: Go's toolchain manager defaults to quietly downloading whatever newer version a go.mod asks for the moment it's actually needed, so go build and go test keep working without you noticing anything happened. Inside Docker it isn't invisible, because the official golang images deliberately turn that automatic downloading off (GOTOOLCHAIN=local), so a build stays reproducible without needing network access partway through. That means the build stage of the Dockerfile from Chapter 4 needs its baked-in Go version raised to match. Update its first line:

FROM golang:1.26 AS build

Everything else in the Dockerfile is unchanged.

Update cmd/localdir-csi/main.go:

package main

import (
	"flag"
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
	mount "k8s.io/mount-utils"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
)

func main() {
	endpoint := flag.String("endpoint", "unix:///csi/csi.sock",
		"gRPC endpoint the driver listens on, as a unix:// URI")
	dataDir := flag.String("data-dir", "/data",
		"directory volumes are stored under, and the health check reads and writes to confirm local storage is usable")
	flag.Parse()

	socketPath, err := driver.SocketPathFromEndpoint(*endpoint)
	if err != nil {
		log.Fatalf("invalid -endpoint: %v", err)
	}

	nodeID := os.Getenv("NODE_NAME")
	if nodeID == "" {
		// NODE_NAME is set by the Downward API in deploy/node.yaml, so
		// this branch never runs in the cluster. It exists purely so
		// `make run`, on your own laptop, still has something sensible
		// to report — your laptop genuinely doesn't have a Kubernetes
		// node name, so falling back to its real hostname is the closest
		// honest answer available locally.
		hostname, err := os.Hostname()
		if err != nil {
			log.Fatalf("NODE_NAME not set, and failed to read hostname as a fallback: %v", err)
		}
		nodeID = hostname
	}

	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: *dataDir},
		nodeID,
		*dataDir,
		mount.New(""),
	)

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	csi.RegisterNodeServer(server, d)
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s (node %s)", socketPath, nodeID)
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

mount.New("") is k8s.io/mount-utils's own constructor for a real, Linux-backed mounter — the empty string is the path to the mount binary it shells out to for some operations, and an empty string tells it to use the default, /bin/mount. The package name is mount, which is why the import gets an explicit mount prefix here even though it's already the default — k8s.io/mount-utils is a fairly generic-sounding import path to leave unlabeled in a file that also has a field named mount on Driver itself.

*dataDir now appears twice in the NewDriver call — once building the health checker, once as Driver's own data directory. That's not duplication to clean up: LocalDirHealthChecker uses it to confirm the directory is writable at all, while Driver uses it to compute per-volume paths. Two different components, two different reasons to know the same root path.

Deploying

make deploy

kubectl get pods -l app=localdir-csi -w

Same 2/2 Running result as Chapter 5 — privileged: true and the new mountpoint-dir mount don't change anything kubelet checks during registration, so nothing about the pod's health should look different yet. What's different is what the container is now allowed to do, which the next section proves.

Proving it, inside the node

Nothing in the cluster calls NodePublishVolume automatically yet — that starts once a real pod requests a volume through a PersistentVolumeClaim, which is Chapter 8's territory. For now, calling it directly is the only way to watch it work, and Chapter 2 already told you where: inside the kind node itself, a real Linux container, the same place these bind mounts actually happen.

grpcurl isn't installed inside the node, so the first step is getting a copy of it there. Rather than guessing at which of several release archives matches the node's own CPU architecture, pull it straight out of grpcurl's own official image — a multi-architecture build that already matches whatever platform your kind node happens to be running on:

docker create --name grpcurl-tmp fullstorydev/grpcurl:v1.9.3
docker cp grpcurl-tmp:/bin/grpcurl ./grpcurl-linux
docker rm grpcurl-tmp
docker cp ./grpcurl-linux csi-dev-control-plane:/usr/local/bin/grpcurl
docker exec csi-dev-control-plane chmod +x /usr/local/bin/grpcurl

csi-dev-control-plane is the actual Docker container backing your cluster's one node — kind names it <cluster-name>-control-plane, and Chapter 2 named the cluster csi-dev.

Chapter 7 will teach CreateVolume to make a volume's data directory automatically. Until then, stand in for it by hand — this is exactly the directory structure CreateVolume will produce later, just created directly instead of through a gRPC call:

docker exec csi-dev-control-plane mkdir -p /var/lib/localdir-csi/data/test-vol-1
docker exec csi-dev-control-plane sh -c 'echo "hello from localdir-csi" > /var/lib/localdir-csi/data/test-vol-1/hello.txt'

Now call NodePublishVolume for real, against the driver's actual running socket, at a target_path shaped like the one kubelet would really choose for a pod. Every grpcurl call so far in this book has called a method that takes no arguments; this one doesn't, so it needs -d, followed by the request as JSON, using the same field names as the .proto file:

docker exec csi-dev-control-plane grpcurl -plaintext -d '{
  "volume_id": "test-vol-1",
  "target_path": "/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount",
  "volume_capability": {"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}
}' unix:/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock csi.v1.Node/NodePublishVolume

{}

An empty response — exactly what the spec says NodePublishVolumeResponse should be, and exactly the same "success looks like nothing" shape NodeGetCapabilities's empty capability list had in Chapter 5. The proof isn't in that response; it's in whether the bind mount actually happened:

docker exec csi-dev-control-plane cat /var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount/hello.txt

hello from localdir-csi

The file you wrote at /var/lib/localdir-csi/data/test-vol-1/hello.txt is readable through an entirely different path — the one your driver was told to publish at, and the one a real container's own filesystem would be pointed at inside a pod. That's the whole chapter, proven with one cat.

Reverse it:

docker exec csi-dev-control-plane grpcurl -plaintext -d '{
  "volume_id": "test-vol-1",
  "target_path": "/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/mount"
}' unix:/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock csi.v1.Node/NodeUnpublishVolume

{}

docker exec csi-dev-control-plane ls /var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/

ls: cannot access '/var/lib/kubelet/pods/test-pod/volumes/kubernetes.io~csi/test-vol-1/': No such file or directory

Gone — NodeUnpublishVolume unmounted it and removed the directory it had created, leaving the original data at /var/lib/localdir-csi/data/test-vol-1/hello.txt completely untouched. Confirm that part too, if you want the full picture:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/test-vol-1/hello.txt

hello from localdir-csi

What you should have now

• internal/driver/driver.go: Driver carrying a dataDir and a narrow, self-declared Mounter interface — satisfied automatically by k8s.io/mount-utils's real mounter, no adapter required
• internal/driver/node.go: NodePublishVolume and NodeUnpublishVolume, both idempotent, both tested against a fake Mounter before either touched real code
• A sidecar-metadata-file pattern (nodePublishMeta, writeNodePublishMeta, readNodePublishMeta) — this book's first, recording what NodePublishVolume actually did so a later call can tell a genuine retry from a real conflict, instead of trusting IsMountPoint's bare true/false alone
• cmd/localdir-csi/main.go building a real mount.New("") and passing it, along with the data directory, into Driver
• deploy/node.yaml: the localdir-csi container running privileged, with a mountpoint-dir volume mounted Bidirectional so mounts made inside the container become visible to kubelet
• Direct, hands-on proof — a file written on one path, read back through a completely different one, inside a real kind node — that a bind mount your own code triggered actually happened

Four of csi.NodeServer's ten methods are done. What's still missing is everything upstream of them: nothing yet creates a volume's data directory, or decides how big it should be, or lets it be deleted. That's the Controller service, and it's next, in Chapter 7.